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IMTRODUCTION TO INSTRUCTIONAL REORGANIZATION SERIES 

Stuart Johnson and Rita Johnson 
Purpose of the Series 

The overall purpose of this series is to assist teachers to 
Improve the effectiveness of their instruction. The procedure 
‘advocated consists of: 

1) breaking course activities into short segments through 
^ich learners can proceed at their ovn pace, 

2) monitoring achievement of learners after con 5 >letion of 
each segment of the course emd, 

3) advising each instructional segment until the desired 
level of achievement is attained by the learners. 

An immediate aim of this series is that you, the reader, after 
vorklng throu^ these self -instructional booklets, vill be able to 
produce a short "instructional package" vhich vill be used vith one 
of your classes and revised until it is effective. The package 
should be self -instructional as far as possible to permit your 
learners to proceed at their ovn pace. 

Specific procedures vill be provided as you complete each 
booklet to assist you in producing the three essential components 
of your package: l) objectives, 2) instructional activities, and 

3) evaluation measures. 

Self -Instructional Materials 

Teachers are potentially the most sensitive, flexible, and 
divergently responsive components of any instructional system. 

Under the limitations of conventional teaching, hovever, they 



seldom have the time nor opportunity to concentrate their efforts 
on that which teachers can do best: l) diagnose individual learner s 

difficulties, 2) interact with learners when they need help on a one- 
to-one basis or in small group discussions, 3) inspire and motivate, 
or 1») identify and encourage creativity and self-direction. Self- 
Instructional packages are essential if we wish to free teachers and 
learners for this type of individualized instruction. However, for 
those who yet wish to capitalize on the advantages of a lecture method. 

It should be noted that the booklets are quite appropriate for use in 
ln 5 >roving lecture-format instruction as veil. 

Trifttructional Objectives 

Some users of these booklets have already developed skill in 
writing objectives so that another teacher can read them and tell 
Vhat the learners are to be able to do by the end of instruction. 

In the event that you are uncertain as to how to write objectives ^ 
in this way, please take time at this point to read and work through: 

Preparing Instructio nal Objectives 

by Robert Mager (Fearon Publishers} • 

If Mager is not available you could use: 

Educational Objectives „ x 

by W. James Popbam (raCET Tape and Filmstrip, fl, UdA} . 

Please remember that these instructional materials have been pre- 
pared for a general audience and you may have to apply their procedures 
to the special problems of your ovn teaching area. When you have fin- 
ished Mager (or Popham) please return to this booklet and continue on 



the folloving page. 



Now that you have practiced writing instructional objectives, 
you may feel that this approach is suitable only for trivial recall 
objectives. This impression has been called the "tragic flaw” in 
both Mager and Popham’s material. They selected simple objectives 
for the purpose of instructional simplicity. However: 



Instructional objectives c^ and should be made 
as complex and subtle as you are capable of 
producing! Your objectives may be primarily 
in the area of student intellectual growth; or 
your objectives may describe desired student 
attitudes . In any case, they must be explicit . 



Use of the Booklets 

A few guideline observations are in order as you begin the 
booklets: 

^Progress in working through the booklets and producing 
your material will be facilitated if you work with a team, 
i.e., one or more of your colleagues. You should use your 
team members as resource persons to critique your objectives 
test items, or instructional activities, and to allay (or 
confirm) any misgivings which arise • 

2. As you work through each booklet, write in the answers 

to each practice exercise. These booklets are yours; anno- 
tate them for later reference. 

3* After completing each of the booklets, refer to the pro- 
cedure sheet on the next page. It is designed to guide your 
step-by-step production of a self -instructional package . 



PROCEDURES FOR PRODUCH^G A 
SELF-INSTRUCTIONAL PAC3CAGE 



After reading each of the booklets, do the folloving; 

1» Book I - Write a set of objectives, including 
one in the affective domain* 

2* Book II - Write your post-test. 

3* Book III - Write an outline of a script for a 

sequence of instruction vhich pro- 
vides the learner ■^7ith the following: 
small steps 
prompts 
practice 

knowledge of results 

4. Book IV - Prepare materials with sufficient directions 

so that learner can work through package 
without additional help from you I 

5. Book V - Administer the package to a few learners. If 

objectives are not achieved, revise package. 
Talking with learners will help locate inad- 
equacies in the package. 

Notes should he kept regarding development of 
the package and the history of its tryout, 
including student comments or other reasons 
for revisions. 



Xou are now ready to proceed to the first booklet. Good luck! 



BOOKTuET I 



SPECIFYING AND MALYZING OBJECTIVES 



Stuart R. Johnson 



RELCV Working Paper 



^ private non-profit corporation^ is a regional education 
lahoratoiy funded primarily by the United States Office of 
Education under provisions of Title IV of the Elementary and 
Secondary Education Act of 1965 * The opinions expressed in this 
paper do not necessarily reflect the position or policy of the 
Office of Education, and no official endorsement by the Office 
of Education should be inferred. 
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SPECIFYING AND ANALYZING OBJECTIVES 



Stuart R. Johnson 

One often hears teachers say (wistfully) ”l’d like Jiiy students 
to learn to think critically and with some originality." Although 
a degree of consensus can sometimes he obtained as to what constitutes 
critical thinking or originality, teachers usually admit that they are 
uncertain about how to produce such learning. Should a teacher begin 
by teaching definitions and terms basic to a field and then proceed 
to concepts and applications? Should a teacher begin by posing a 
problem, then filling in terms and concepts as they are needed? 

Should one teach chronologically? Inductively? Deductively? 

It is clear that any answers to these questions will require 
attention to more fundamental questions. What are the different 
kinds of tasks in which the teacher wants his students to develop 
proficiency? What prerequisite tasks are involved and how should 
the learning activities be sequenced? 

The purposes of this booklet are: 

1. To develop in you the ability to apply rigorous cri- 
teria in determining the desired aims of instruction, 
and, 

2. To assist you in analyzing the learning tasks which 
underlie a desired instructional outcome. 

The exercises in this booklet have been prepared with several 
assiimptions in mind: 

1. The reader has accepted the importance of developing ob- 
jectives as a beginning point in instructional planning. 



2. Instructional objectives should be stated in terms of vhat 
nev things the students can do as the result of instruction# 

3# The reader is applying what is learned from these pages to 
the development of a specific instructional plan, and de- 
cisions made in one stage of planning will be modified hy 
decisions based upon later considerations. 

You have, of course, mastered the essentials of the subject 
matter or skills which you teach. However, you may not have thor- 
oughly analyzed or classified the kinds of learning tasks involved. 
Presumably you are interested in developing and implementing an 
instructional system" for a particular group of learners with 
known (or assumed) beginning competencies. The comments which follow 
id.ll appear more pertinent if you can refer to your own first draft 
of objectives for a proposed instructional plan. 

The objectives making up this first draft 17111 probably need 
to be broken down into more specific descriptions of learner behavior. 
You should also employ some scheme designed to assure that all the 
learner behavior being sought has been included. Finally, given an 
initial set of objectives, you must be sure that relevant prerequi- 
site skills are taken into account in your instructional plan. 

Two forms of logical analysis will be used: 



Analysis b y Type of Behavior means determining where 
fiiven objective fits with respect to the major 
categories of learning outcomes, that is, psychomotor 
(skills ), affective ("attitudes"), and cognitive 
("knowledge"). 



Task Analysis means breaking down a complex learning 
task into successively simpler sub-tasks, each of 
which is necessary for the final accomplishment of 
the complex task. 



THREE MJOR TYPES OF EDUCATIONAL OBJECTIVES 

All educational objectives can be classified as primarily in 
one of three ’’domains” or large categories: psychomotor, affective, 

or cognitive . An example of a psychomotor learning objective is 
sinking baskets with a basketball; the objective involves controlling 
the muscles and making the proper motions so the ball Td.ll regularly 
go through the hoop. Objectives in the affective domain involve 
feelings and attitudes. An example would be the learner’s attitude 
toward a subject following the first day’s introduction by the instruc- 
tor. The objective would be a favorable feeling toward the subject or 
a desire to learn more about the subject and the benefits that can be 
derived by mastering the subject. Cognitive learning objectives in- 
volve intellectual processes including remembering, understanding, 
problem-solving, and other kinds of learning which involve informa- 
tion storage, retrieval and processing. 

Most instruction includes some learning in all three of the 
domains. Often, the cognitive aspects are most emphasized but cogni- 
tive learning will also depend upon affective learning and psychomo- 
tor skills. For example, the student who has never learned to pay 
attention or "attend" (affective behavior) to a lecture may have 
very little information to process and store (cognitive) from an hour 



of listening to instruction, particularly if his penmanship (psychomo- 
tor) is so erratic that he cannot review his notes . 



Each of the following five statements shows instructional concern 
about behavior classified primarily as psychomotor, affective, or 
cognitive. Label each statement with one of these three terms in 



the space to the right: 



1 . 

2 . 

3. 

h. 



The student *s interest is an important 
factor for success in ny class. 

In reading a micrometer, the student must 
get the proper "feel” as he turns the knob 

The important thing is to understand what 
you read, not to just remember what was 
in the chapter. 

Tighten the nut until it is "finger 
tight." 




5 . Before you start the job, think through 
the steps in the procedure. 



Answers should read: 



1. Affective 

2. Psychomotor 
3* Cognitive 

Psychomotor 
5 . Cognitive 



If you had difficulty with the classifications, please review 
the paragraph which describes the three categories of learning ob- 
jectives. You might also wish to look over the fuller descriptions 
of each classification in the appendices. 

You have now taken a first step in the analysis of type of 
learner behavior; classifying examples into one of the three large 
domains or categories. Various specialists also have attempted to 
analyze educational objectives within each of the three domains. 



A CLASSIFICATION OP PSYCHOMOTOR OBJECTIVES 



The performance of responses requiring muscular coordination 
(the psychomotor category) has been analyzed by Elizabeth Simpson. 
Her classification is abstracted below and is presented in greater 
detail in Appendix A. She proposes that the five categories are 
arranged according to increasing complexity, and that the first 
step is necessary to the completion of the second and so on. 



Below, decide which objective is more complex and place a check 
by the more coEplex objective: 



1. Student shall grasp the tennis racquet handle 
as if he were shaking hands with it. 



2, When hitting the ball, student shall not let 
the racquet head drop below the handle • 



In the exercise above, holding the racquet properly is a neces- 
sary'' preliminary to hitting the ball with the racquet in proper 
position; therefore, the second objective is more complex. 

Now please examine the psychomotor classification \diich follows: 



THE PSYCHOIOTOR CLASSIFICATION 



lEAST © Perception - becoming aware of a situation which may 
COMPLEX result in a muscular response. 



© Set - adjustment or readiness for a particular kind 
of action. 



® Guided response - imitation of another person per- 
forming an act or trying various responses in trial 
and-error fashion. 



0 Mechanism - an habitually correct response. 



most ® Complex overt response - performing a complex act 
COMPLEX correctly with ease and without hesitation. 
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At vhat level of the PSYCHCKOTOR CIASSIPICATIOW is each of the 
objectives below? 



1 . 



2 . 



Student shall grasp the tennis racquet 
handle as if he were shaking hands with it. 

When hitting the hall, student shall not 
let the racquet head drop below the handle . 



Objective #1 should have been labeled "Set" and Objective ^ 



labeled "Guided Response" or higher depending upon whether the student 



is beginning to gain this skill or is displaying the skill on a habit- 
ual basis . Objective #1 is considered to be the physical readiness 
necessaiy to hit the ball, and is therefore, placed in the "Set" 
category. If you need to develop more elaborate sets of psychomotor 
objectives, it is likely that Appendix A will be helpful since it 
contains more complete descriptions of each category. 



A CLASSIPICATIOR OP APEECKVE OBJECTIVES 



The affective category has been analyzed by a group of measure- 
ment e:q)erts . Their classification is presented in Appendix B and 
a modified version follows below. If you and your colleagues are to 
design an instructional system which leads to predictable achievement 
by learners, you should not ignore so in^ortant a consideration as 
the attitudes and feelings of the learner. It may be that something 
^ effective category will make the difference between failure 
and success on a particular cognitive objective . The affective clas- 
sification is arranged so as to show increasing levels of personal ac- 
ceptauice . 






J 



THE AFFECTIVE CLASSIFICATION 

LOW o Receiving or Attending - avareness (of form, color, 

ACCEPTANCE 

differing viewpoint s^, importance of something), willing- 
ness to receive (as shown hy hearing viewpoints of 
others or accepting differences), and controlled or 
selected attention (as shown hy listening with discrimi- 
nation or sensing Importance of details) 

© Responding - acquiescence or compliance, , willingness 
to respond, and satisfaction in j’CQKDnse (as shown hy 
e3q>ressing pleasure) 

© Valuing - acceptance of a value, preference for a value, 
and commitment or conviction (as shown by being loyal 
or attempting to influence others) 

© Organizing - developing a value system (as shown hy 
identifying the characteristics of something valued or 
hy making plans concerning social problems) 
Characterization - developing an ethical code or a 
philosophy of life (as shown by consistent behavior) 

This outline of affective terms arranges in sequence the feelings 
and the emotional responses which are usually labeled "interest,” 
attitudes, appreciation,” and "values*” The range is from simple 
attention to the development of a value system# The two lower levels, 
receiving or attending” and "responding,” are basic to all successful 
instruction. The higher levels are concerned with personal and inter- 
personal adjustments, values formation and character development. 



HIGH 

ACCEPTANCE 



o 

ERIC 



■„■■.. ^■■;rf>.^..^.’t^ ... 
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See if you can judge which two of the following objectives are 
lowest in terms of acceptance; place an "L" in the blanks. Then 
decide which two are highest in acceptance and place an ”H" in the 
blanks: 

1. Student reads newspaper editorial on freedom f— 

of speech when assigned as homework. 



2. Student defends a fellow-student’s right to 
advocate governmental censorship of news. 



3* Student listens to what is said during 
class discussion on freedom of speech. 

4. During class discussion student describes 
the history and underlying assumptions in 
his advocacy of free speech. 

You should have placed an "L" by number 1 and 3; an ”H” should 
appear by number 2 and 4. If your answers did not correspond^ it 
may be helpful to review the affective classification in the appendix, 
^y way of examples, the four objectives above are in the following 
categories: 

1. Desponding 

2. Characterization 

3- Receiving 

4. Organization 

It is obvious that affective objectives are Important to instruc- 
tion. For example, it is hard to imagine a successful instructional 
program which failed to capture the student’s attention or failed to 
make him willing to respond. Furthermore, it is the writer’s experi- 
ence that most instructors care deeplj’’ about developing some sort of 
positive attitude among students toward the subject of instruction. 
Pure cognitive learning without e:q)ression in the real-life of the 
student is a futile exercise. 




' 



A CLASSIFICATION OF COGNITIVE OBJECTIVES 

A practical classification of cognitive learning objectives was 
provided by the same group of testing experts who analyzed affective, 
learning. It is arranged in terms of complexity. Its chief value to 
instruction is in helping to call the teacher* s attention to more com- 
plex objectives than mere memorization and recall. The complete clas- 
sification system has been summarized below from the original which is 
found in Appendix C. You should also review the original at some future 
time for examples and further e^qplanation . 

AN ABBREVIATED CLASSIFICATION OF COGNITIVE OBJECTIVES 

LEAST 1 .0 Knowledge or ability to recall information (memory) 
COMPLEX 

1.1 Knowledge of Specifics - facts, definitions, 
symbols 

1.2 Knowledge of Ways of Dealing with Specifics - 
of forms and conventions, steps in a process, 
categories, etc. 

2.0 Comprehension or understanding 

2.1 Ability to translate or rephrase 

2.2 Interpretation, or recognizing essentials 

2.3 Extrapolation, or recognizing implications, 
limitations 

3.0 Application or transfer - ability to use knowledge 

and understanding in a novel situation to solve 



problems 



4.0 Analysis or “breaking a vhole into its elements, analysis 
of relationships or organizational principles 
5*0 Synthesis or putting together elements and parts to 
form a new whole 

MOST 6.0 Evaluation or making judgments in a field using 

COMPLEX 

internal evidence or external standards 
Achieving coB 5 >etence at any given level in the cognitive classifi- 
cation is thought to require competence at successively lower levels. 

For example, to e:q>lain the meaning of a word hy saying "in other 
words” (2.1) requires memory of the given definition (l.l). 

Tiy classifying the following instructional objectives into one 
of the following levels: Knowledge, Comprehension, or Application. 

Write ”K, ” ”C, ” or ”A" in the spaces provided: 

1. To give the appropriate English equivalents j j 

of forms of greetings in French, hut not 
exact translation. 

2. Given the names of three contemporary com- 
posers student shall give the name of one 
composition hy each 

3* Student can explain in his own words the 
meaning of a political cartoon. 

4. Student shall convincingly represent a 
memher of a tax-payers* association during 
a. mock meeting of the city council . 

The intended answers are: 

1. Comprehension 

2. Knowledge 
3 * Comprehension 
4. Application 




We can beet provide the conditions for learning idien we under- 
stand the nature of the learner's tasks. These depend, in turn, upon 
the level of complexity of the objectives. If the learning task in- 
volves Mmorjr only, we can provide ways to help the student remember 
and provide practice sessions in remembering. If the learning task 
Involves comprehension of an idea or the meaning of technical terms, 
something more than memory is involved, for memory alone will not 
help the learner to "explain in his own words." Students will need 
practice at rephrasing ideas or concepts in order to develop competence. 

Similarly students should be given practice at~whatever cognitive level 
”we seek competence. 

SUMMARY OF MiiLYSIS OF TYRE OP BEHAVIOR 
You now have three frameworks for analyzing learner Behavior. 

You have seen that all objectives can he classified primarily into 
one of three categories; psychomotor, affective, or cognitive. The 
affective domain has been divided into five classes: receiving or 

attending, responding, valuing, organizing, and characterizing be- 
havior. The cognitive category has been subdivided into six classes: 
knowledge or memory, comprehension, application, analysis, synthesis, 
and evaluation; some of the classes have been subdivided further. A 
psychomotor classification scheme describes muscular responses at 
five different levels of complexity; perception, set, guided response, 
mechanism, and complex overt response. 

WHAT IS TASK ANALYSIS? 

We have been looking primarily at the type of behavior which the 
learner is to develop. We now change perspective to examine the 



sequences of the learner tasks as he practices for mastery of the 
“behavior. We are looking at the same objectives but from a differ- 
ent viewpoint. Since analysis means "breaking down into simpler 
elements, we shall examine learning tasks and seek to identify sub- 
tasks upon which mastery of the terminal task depends. 

Task analysis is essential in a systems approach to instruction: 
it makes mastery of a subject possible by identifying small, compo- 
nent, learner steps each of which is possible to master with adequate 
learner practice . 

For example a person who has never seen the pilot *s compartment 
of an airplane can learn to fly the plane if the process is broken 
down into small enough steps, properly sequenced, and each step mas- 
tered before proceeding to the next. Faulty task analysis or im- 
proper sequencing can lead to learning failures . 



Task analysis means breaking down a learning (which 
is identified by an objective) into component tasks, 
each of which must be mastered as a prerequisite to 
mastery of the total task. 

To analyze a learning task, we begin with a statement of the 
terminal objective. When phrased in terms of the learner behavior 
the objective tells what the learner will be able to do following 
the instruction. To analyze the overall task, we ask such questions 
as: What must the learner be able to do to achieve the objective? 

What kinds of learnings are involved? What prior skills are neces- 
sary? What specific knowledge is required? What concepts or mean- 
ings must be understood? What is prerequisite to ultimate success? 
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In cognitive learning, the cognitive classification suggests the 
sequence in which learning tasks could be presented. For example, in 
order to apply a principle in a new situation, the learner must under- 
stand (be able to explain) the principle and he must recall acceptable 
words and phrases to use in speaking or writing about the principle . 

However, most learning tasks include a number of subtasks. Let’s look 
at an example. 



AN EXAMPLE OP TASK ANALYSIS 

Suppose an objective in an English class were stated like this: 

"to be able to write a one -page expository paper 
supporting a political point of view." 

What is the nature of the task? Since our frame of reference is college 

English, we may assume the student already possesses the psychomotcr 

skills of handwriting or typing and ignore these in the task analysis, 

or we may wish to add them as objectives . In addition, the student must 

be able to tell the difference between a paper "about" some topic or 

political viewpoint and one which "supports" a political point of view, 

before he can write such a paper. The learner who has mastered this 

subtask (by practicing this specific kind of discrimination) has a 

clearer view of his goal and of his assignment. 

What else is required to write an expository paper? To say that 
a paper is made up of paragraphs and that paragraphs are made up of sen- 
tences, and the abilities to compose each are subtasks, is to over- 
simplify the task. To illustrate, one subtask is to write the introduc- 
tory paragraph - which is different from other types of paragraphs. 

What is required in order to write a good introductory paragraph? If 



o 



"arouse interest" is one of the requirements, vhat does interest 
arousal require? Does it involve choice of wrds, length of sentences 
a particular type of sentence, or the arrangement of sentences in the 
opening paragraph? If the learner *s task (or suhtask) is "problem- 
solving” (the problem being to write a paragraph which will arouse 
interest), what are the steps in solving the problem and what are the 
requirements at each step? 

On the next page is an example of a simple task analysis, one of 
many tha,t could be logically derived from the previous ohject5,ve. 



TASK ANALYSIS: ^ Write a One -Page Expository Paper 



Subtask 1: To discriminate between a '’-well -written" paper and one 

which does not meet the instructor *s criteria. 

Subtask a: To remember all of the instructor *s criteria, or to 

review the list supplied asking successful questions 
about unclear criteria. 

Subtask (l): To find out what the criteria are, if the previous 

list is misplaced- — 

Subtask 2: To write an introductory paragraph which arouses interest. 

Subtask a: To discriminate between paragraphs which arouse inter- 

est and paragraphs which do not do so, with particular 
readers • 

Subtask b; To identify the elements of a paragraph which arouse 
interest. 

Subtask c: To select "arousal" words and phrases. 

Subtask d: To write an introductory sentence which attracts 

attention . 

Subtask (l): To discriminate between sentences which at- 

tract attention and those which fail to do so . 
Subtask e: To evaluate the paragraph as written, using the 

criteria of a good introductory paragraph and/or to 
elicit the evaluation of peers or instructional 
staff. 

Subtask 3: To write a paragraph which supports the viewpoint. 

Subtask a: To discriminate between paragraphs which support a 

viewpoint and paragraphs "about" a viewpoint. 



“v, 4 . « 






Obviously each of the subtasks implies other subtasks* 



Where you stop in the derivation of subtasks is 
a function of where you choose to have your 
instructional responsibility begin. 
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For a practice exercise in task analysis, let us consider a straight- 
forward problem in training a grocery checker to calculate the individual 



price of "group-priced” items: 



Objective: 

Within ten seconds, student can correctly calculate the 
cost of one grocery item, given the cost of several such 
items, for example, 2 / 67 p, V254» 

Please list two subtasks which are prerequisite to attainment of 
this objective: 

Task Analysis: 



1 . 

2 . 

To determine the adequacy of your task analysis, it is likely that 
tiro important subtasks are: 

1. Student can correctly divide and roundoff 2-digit numbers by 
the numbers 2 through 9^ or, 

2. Student can correctly multiply 2-digit numbers by 1-digit 
numbers . 

More fundamental subtasks might include: 

a. -student can state that "any fraction of a cent should raise 
the cost of the item to the next higher whole cent," or 

b. -student can translate the symbol " 2 / 67 f^" into the statement 

two of these items cost 67 js'." 




To further validate a list of subtasks you should consult a fellow 
teacher and obtain his agreement that your subtask items are prerequi- 
site to the objective. When you are attempting this process in your 
ovn teaching, a colleague is a reasonable source to validate your task 
analysis • 

SUMMARY OF TASK AKALYSIS 

Success in any task depends upon success in the prerequisite 
tasks, i.e., upon attaining the prerequisite skills, knowledges, and 
understandings. Therefore, prerequisites for any task must be iclentified 
The learner must have previously acquired them or be provided practice in 
them at the time they are needed, ^e learner’s success in any task de- 
pends, in part, upon the instructor’s planning: upon identifying the 

prerequisites, upon determining whether the learner does (or does not) 
have them, and upon providing assurance that the learner has mastered 
them. A "good." task analysis therefore includes subtasks which identify 
the kinds of learnings that are essential for attainment of the objective 
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OBJECTIVES FOR BOOKLET I 
SPECIFYING AND ANALYZING OBJECTIVES 



Ob.lectives for this booklet; 

1» After completion of this booklet, the participant sha3.1 cor- 
rectly categorize a set of instructional objectives into the 
psychomotor, affective, and cognitive domains. 

2. Given two psychomotor objectives, the participant shalJ. cor- 
rectly select the one which is highest in complexity. 

3* The participant shall correctly rank three affective objectives 
in terms of whether the objective is high, medium, or low in 
the affective classification. 

The participant shall correctly classify four cognitive objec- 
tives into the following categories! Knowledge, Comprehension, 
Application, and Analysis. 

5* Given an instructional objective, the learner shall select from 
a list of subtasks those which eire prerequisite to the objective. 

Now complete the criterion test in order to determine whether this 
booklet has been effective and whether the objectives for this booklet 
have been achieved. 



CRITERION TEST - BOOKLET I 



Place a ^ ^ or C by each objective depending upon whether it is 
nrimarily Psychomotor, Affective, or Cognitive in nature: 

1. Student shall draw an inked line with no discernible 
deviation in width. 



2. Student can correctly select the phrase describ 
ing 10 common map symbols. 

3* Student can summarize the arguments of 2 peers 
to the satisfaction of each. 



Student will request that campus discipline 
regulations be revised by a student committee. 

5* Student can apply band-aid without impairing 
sterility of the pad. 



6. Fewer than 10^^ of the students shall drop a 
class during the first 3 weeks . 



o 



Read the next 2 psychomotor objectives and place 
an X after the objective vhich is more complex: 

7 • Student shall fill the hypodermic syringe -with- 
out admitting air bubbles into the syringe* 

8* Student shall sloT/ly advance the hypodermic 
needle into tissue near the surface of the 
-wound, maintaining constant pressure on the 
syringe plunger. 

In the following 3 objectives, -write "High" by the 
objective which is highest on the affective hierarchy 
and write Low" by that which is lowest: 



9* Student states that watching an assigned TV 
documentary program -was interesting. 

10. Student watches a film in class. 



11. Student searches TV Guide for programs on 
science for his leisure vie-wing. 



One example of a cognitive objective at each of the 
levels Knowledge, Comprehension, Application, and 
Analysis is given below. Place the letters "K " "C “ 
Ap, and An in the appropriate blanks: 

12. In his own -words, student -will state the 
meaning of the term "social system." 



^3* After observing a public picnic ground, the 
student shall decide which social subgroups 
were present^ he shall give them descriptive 
names, and shall state observations to defend 
his classification. 



Ih. Students will correctly define the terms 
status" and "hierarchy." 



15* Given (l) an elementary school class roster, 
and (2) the name of the pupil selected by each 
pupil as "my best friend in class," student 
will identify the most popular pupils and the 
social isolates. 



Place a check by those subtasks which are necessary to 
the accomplislmient of the objective: 

Objective; Student shall s-wim 50 yards using breast stroke 
Subtasks: 

l6. Student can locate small object on bottom of pool. 



17* Student must tread water for 15 seconds. 

18. Student shall coordinate arm stroke and kick. 

19 . Student shall coordinate breathing and arm 

strokes. 

20. Student shall shower before getting into public 

pool . 

NOIE: 

After you have checked your answers to the criterion tes 
please turn to the last page in this booklet. 



ANSWERS TO THE CRITERION TEST: 



P 



C 



c 



A 



P 



A 



X 



IiOV 

—High 

C__ 

An 

K 

• An 



X 

X 



2 . 

. 3 . 

h. 

. 5 . 

6 . 

. 7 . 

8. 

. 9 . 

. 10 - 
, Il- 
ls. 

13 . 

14 . 

15 . 

16 . 
IT. 
18 . 

19 . 

20 . 



Minimum performance for this booklet; 90 % of 

the participants should get 90 % of the criterion 
items correct. 
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APPEiroiX A 

?he classification of e ducational objectives 

PSYCHOMOTOR DOMAIN 

1.0 Perception - the essential first step; the process of becoming 
aware of objects, qualities, or relations by way of sense organs. 

1.1 Sensory stimulation - the impingement of a stimulus upon one 
or more of the sense organs. 

1.11 Auditory 

1.12 Visual 

1.13 Tactile 

1.14 Smell 

1.15 Kinesthetic 

1.2 Cue selection - identification of the cue or cues and associ- 
ating them with the task to be performed. 

1 .3 Translation - relating of perception to action in performing 
a motor act; the mental process of determining the meaning 
of the cues received for action. 

2.0 Set - a preparatory adjustment or readiness for a particular kind 
of action or experience. 

2.1 Mental set - readiness in the sense of having made the anatom- 
ical adjustments necessary for a motor act to be performed. 

2.2 Emotional set - readiness in terms of attitudes favorable to 

the motor act's taking place. WiUingness to respond is implied. 

3.0 Guided response - an early step in the development of skill. Empha- 
sis is upon the abilities which are components of the more complex 
skill. 

3.1 Imitation - the execution of an act as a direct response to 
the perception of another person performing the act. 



3 .2 Trial and error - trying various responses, usually with some 
rationale for each response, until an appropriate response is 



achieved. 

4.0 Mechanism - the habituation of a learned response. At this level, 
the learner has achieved a certain confidence and degree of skill 
in performance of the act. 

5-0 Complex overt response - the individual can perform a motor act that 
is considered complex because of the movement pattern required; a 
high degree of skill has been attained; the act can be carried out 
smoothly and efficiently. ^ 

5*1 Resolution of uncertainty - the act is performed without hesi- 
tation; the individual knows the sequence required and so pro- 
ceeds with confidence. 

5 .2 Automatic performance - the individual can perform a finely 

coordinated skill with a great deal of ease and muscle control. , 



From: Simpson, Elizabeth Jane, The Classification of Educati onal Objec- 

tives, Psycho moto r Domain . Urbana, Illinois; University of 
Illinois, 1966. (Unpublished project report.) 


















% *.“' jA W *^' »• ’ 






2k 



APPENDIX B 

TAXONOMI OP EDUCATIONAL OBJECTI-TES 
APEECTIVB DOMAIN 



1.0 Receiving (Attending) 

1.1 Awareness (conscious of a situation, object, stage of affairs) 

1.2 Willingness to receive (giving attention but neutral toward 
the stimulus) 

1.3 Controlled or> selected attention (selection of stimuli to be 

attended to: attention controlled by -the learner) 

2-0 Responding 

2.1 Acquiescence in responding (compliance or obedience) 

2.2 Willingness to respond (voluntary response: proceeding from 

one*s own choice) 

2.3 Satisfaction in response (behavior accompanied by a feeling 
of pleasure, zest, or enjoyment) 

3.0 Valuing 

3*1 Acceptance of a value (shown by consistency of response to the 

class of phenomena with which a belief or attitude is identified) 

3-2 Preference for a value (sufficient commitment to a value so the 
individual will pursue, seek out, or want it) 

3*3 Commitment (belief involves a high degree of certainty border- 
ing on faith; includes loyalty to a position, group, or cause; 
shown by efforts to convince others) 
k,0 Organization 

4.1 Conceptualization of a value (shown by attempts to identify 
characteristics of an object or position valued and by expres- 
sion of Judgments about a value) 



o 
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4.2 Organization of a value system (bringing together a complex of 
values into an ordered relationship vith one another) 

5.0 Characterization by a value or value complex 

5.1 Generalized set (the individual acts consistently in accordance 
vith the values he has internalized) 

5.2 Characterization (having developed a consistent philosophy of 
life or a code of behavior vhich becomes characteristic of the 
Individual) 



Adapted from: Krathwohl, David R., et Taxonomy of Educationa l Ob.le c- 

tives. Handbook lit Affective Domain . Nev York: 'David 
McKay Co., 1964. 
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APPENDIX C 



TAXONOMY OF EDUCATIONAL OBJECTIVES 
COGNITIVE DOMAIN 



1.00 Knovledge (Recall of Specifics, Pattern, Structure, etc.) 

1.10 Knowledge of Specifics (specific bits of information) 

1.12 Knowledge of Terminology 

1.20 Knowledge of Ways and Means of Dealing with Specifics 

(Organizing) 

1.21 Knowledge of conventions (usages, styles, practices, 

forms) _ 

1.22 Knowledge of trends and sequences (with respect to 
time) 

1.23 Knowledge of classifications and categories (arrange- 
ments, classes) 

1.24 Knowledge of criteria (judging facts, principles, 
opinion, criteria) 

1.25 Knowledge of methodology (techniques, methods of 
inquiry) 

1.30 Knowledge of Universals and Abstractions (theories and 

generalization) 

1.31 Knowledge of principles and generalizations (particul; 
abstraction) 

1.32 Knowledge of theories and structures ( body of princi- 
ples, range of specifics systematic view) 

Intellectual ABILITIES and SKILLS 

2.00 Comprehension (relating knowledge to other material or seeing the 

full implication) 

2.10 Translation (paraphrasing or restructuring ideas) 

2.20 Interpretation (summarization, reorganization) 

2.30 Extrapolation (extension of trends beyond given data) 

3.00 Applicati on (use of abstractions in concrete situations) 



li.OO Analysis (the break ing-doiwi of information into its elements) 



4.10 Analysis of elements (distinguishing facts from hypothesis^ 
etc.) 

4.20 Analysis of relationships (connections and interactions of 
parts of a structure of knowledge) 

4.30 Analysis of organizational principles (organizational system- 
atic arrangement) 

5.00 Synthesis (putting together of elements and parts to form 
structure) 

5.10 Production of a unique communication (communicating to other?:) 
5o20 Production of a plan or proposed set of operations 

5.30 Derivation of a set of abstract relations (formulating 
hypotheses or propositions) 

6.00 Ev8.1uation (quantitative and qualitative judgments, using standards 
of appraisal) 

6.10 Judgment in terras of internal evidence (logical accuracy, 
internal consistency, etc.) 

6.20 Judgraent in terms of external criteria (evaluation of in- 

4 

ternal data to outside influences and selected criteria) 



Adapted from: Bloom, et al . , Taxonomy of Educational Ob.lectives, 

Handbook I. Cognitive Domain . Nev York; David McKay Co .. , 

1956. 









A % 









EEVISION DATA SHEET - BOOKLET I 

Please complete this sheet and mail to: 

The Johnsons 

Regional Education Lahoratory 
for the Carclinas and Virginia 
Mutual Plaza 

Durham, North Carolina 27701 



Instructions: 

As you work through these booklets you will realize that the 
authors are committed to the use of achievement data to revise 
Instructional materials and procedures. 



Please review your answers to the practice exercises in the 
booklet. Mark below any exercise where your answer did not 
match the "correct” answer. By doing this you will help us 
identify areas in the booklet where we have failed to provide 
adequate instruction. Please feel free to comment on your 
reasoning or sources of confusion for any exercise which you 
missed (i.e., we failed). Also, mark any criterion test item 
which you missed. Be assured that this revision data sheet 
reviewed by us with intense interest! Your responses 
will guide our revisions. 



Practice Exercises: 
Page 

4 . 1 . 

2 . 

^3. 

4. 

5- 



8 . 1 . 

2 . 

^3. 

4. 

10 . 1 . 

2 . 

^3. 

k. 

l6. 1. 

2 . 



The Johnsons 



Criterion Test: 
Page 

18. 1. 

2 . 

3- 

4 . 

19. 5. 

6 . 

7,8 

9- 

10. 

11 . 

12. 

13. 

l4. 

15. 

20. i6, 

17 - 

18. 

n-9.‘ 

20. 
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MEASURING ATTAINMENT OF OBJECTIVES^ 

Stuart R. Johnson 

Obtaining Evidence of Learners* Achievement 
of Instructional Objectives 

Systematic approaches to the development of instructional pro- 
grams' invariably stress the need to define the goal or objectives 
in terms of student behavior. Prom the behaviorally stated objec- 
tives it is possible to derive specific measuring procedures ("cri- 
terion measures") to obtain evidence of program effectiveness. Re- 
gardless of the type of desired outcome of any system, information 
is needed regularly as to whether the predicted outcome (i.e., ob- 
jective) is gradually being accomplished. Obtaining this informa- 

• * « 

tion is referred to as a measurement process; the process consists 
of two steps ... some sort of observational comparisons are necessary 
as well as a means to record the observations. 

For this paper it is important to note three 

separate possible uses of measurement data. In an instructional 
setting, these three uses may be illustrated as follows: 

1) Measurement data may be fed back immediately to the learner. 
This instructional practice is referred to as "reinforcement” 
for correct responses or giving "knowledge of results," as im- 
plied when directions are given to the learner as to his next 
steps (review, restudy or continue on to new tasks) . 

^Some of the procedures are described more fully in a most helpful and 
entertaining paperback: Webb, Eugene J. , ajL. , Unobtrusive Measures 

Monreactive Research in the Social Sciences . Chicago: Rand McNally 

& Company, 1966. 



2) Measurement data may be used to accept or reject an in- 
structional system. For example, a teacher might conclude 
on the basis of poor learner performance that a given work- 
book was unsatisfactory and should not be used again. 

3) Measurement data may be used in order to improve an in- 
structional procedure or system. 

It is this third and last consideration that guides the com- 
ments in this paper. Although the first two uses are are perfectly 
appropriate uses for measurement data, this paper deals with measure- 
ment procedures especially for the purpose of upgrading the effective* 
ness of the instructional system through repeated revision. 

In the previous booklet ("Specifying and Analyzing Objectives") 
you have undertaken the development of a variety of instructional 
objectives. In this exercise you will develop a rationale for mea- 
surement procedures which provide evidence of learners * achievement 
of objectives. 

After the completion of this instruction you should be 
able to recognize a number of measurement procedures which 
can serve as criterion measures for specified instructional 
objectives . 

In addition you should be able to distinguish product 
from process criterion measures as well as distinguish reactive 
from non-reactive measures. 

You should be able to distinguish minimum performance 
■ standards from either instructional objectives or their 
criterion measures. 



CRITERION MEASURES 



Once a set of instructional objectives has been selected, 
specific procedures must be developed to gather evidence as to 
whether the objectives have been met. Problems of confusion and 
learner misdirection will arise if these evidence gathering pro- 
cedures ar.e not made specific prior to instruction. For example, 

consider the following objective: 

The learner shall commit himself to a position with regard 
to Fair Housing. 

There is jno way to know, . (when later trying to gather evidence of 
student learning) whether the instructor wanted the learner to: 

•sign a petition, 

.write a letter to a legislator, 

•write an essay on Pair Housing, 

, raise objections in class discussion about unfair 
housing practice, 

.ask for the address of a local ACLU chapter, 

.volunteer to head up a committee on Fair Housing 
practices, or, 

• simply state in a paper *and pencil test that he favored 
Fair Housing legislation. 

Any of the above behaviors might reflect commitment. For this 
reason, it is important that the instructor makes explicit in his 
plans (in advance of instruction) some of the specific criterion 
measures he will use to determine whether the objective has been 

achieved. 

Criterion measures allow the collection of evidence of 
change in behavior, thus giving evidence of instruction— 
al effectiveness. — 

o 
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Examples of criterion measures might include any procedure 
which will* yield a numerical index of perfoinnance ; 

.a multiple- choice or true-false test with scoring key, 

^a rating scale or scoring sheet for essay questions, 
constructed response, term papers, book reports, etc., 

% 

.a check sheet, rating scale, interview schedule, or 
observational record form to measure a student *s verbal 
or psycohomotor performance, 

.counts or tallies of types of observed behavior. 

In order that you may be completely clear about the relation- 
ship between objectives and criterion measures, examine the following 
objective and criterion measures, and mark that alternative which de- 
scribes a criterion measure for the objective: 

Objective: The learner will recognize examples of in- 

ductive reasoning. 

_ Alternative A. Learner underlines those sentences in a 

_____________ newspaper editorial which constitute an 

inductive argument. 

Mternative B. Learner reads the textbook description of 
deductive vs. inductive reasoning. 

Alternative C. Learner writes short essay in which he 

* ' describes the essential characteristics of 

inductive reasoning. 

Alternative A describes a situation wherein the learner "rec- 
ognizes examples of inductive reasoning," the behavior called for in 

■ 

the objective.. By contrast. Alternative B describes, an activity 
which assists the learner in achieving the objective, but it does 
not result in an index of the learner’s ability to perform t he be- 
havior described in the objective. Alternative C would give evidence 
of the learner's descriptive capabilities, but, again would not tell 
whether the learner could "recognize" examples of inductive reasoning. 

er|c 
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Only Alternative A describes a criterion measure which will yield 

S ' 

' a numerical index of performance > 

\ If you are still a bit uncertain about criterion measures, try 

I one more example. Which alternative describes a criterion measure 
^ for the objective shown? 

^ Objective; Student will calculate prices of items of 

merchandise which are to be discount-priced 
during a "1/3 Off" Sale. 

Alternative A. Learner explains the procedure by which 

percentages are calculated. 

Alternative B. Learner prepares price tags for sale items 

originally priced at $1.98 and other com- 

... * mon prices. 

Alternative C. Learner assists parent in choosing a lower 

priced product from alternatives on sale 
in the supermarket. 

The correct answer is "B" since the objective indicates that 
the student will " calculate" rather than simply describe a proced- 
ure for making the calculations. The criterion measure (in this 
instance) is the number of correctly completed sales price tags, 
i.e. , the numerical index of performance would be the number of 
accurate pricings out of the total possible (in the form of correct- 
ly filled-out price tags) . 

Instructors often focus exclusively upon the use of paper-and*" 
pencil tests as criterion measures without taking advantage of other 
; types of evidence gathering procedures. In order to assure that a 
1 broad variety of criterion measures have been considered, two dif 
[ ferent schemes are suggested. 

I . • . 

I .The use of product ys. process criterion measures. 

•The use of reactive ys. non-reacl^ve criterion measures. 

I ' • 
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PRODUCT-PROCESS CRITERION MEASURES 
Th0 product— procGss distinction is quite straightforward. A 
-teacher might express the distinction by saying: "I can assess a 

student's performance by watching him in action, or by examining 
what he makes.” This idea underlines the distinction between "pro- 
duct" and "process" measures: 



.Product measures give some tangible evidence of a learn- 
er's perfomance which can be stored or filed for later 
reexamination if desired. 

y 

.Process measures allow the collection of evidence of a 
learner's performance only as it occurs . 



r 
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mich of the following is a product measure? 

"N 

.A procedure which measures what learners have constructed. 
.A procedure which measures what learners are doing. 



The first statement describes a product measure; the second 
describes a process measure. 

The examples (below) may help to further clarify the product- 
process distinction: 



O 
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Product Measures 

Procedures for comparing: 

-any written responses such 
as those on tests, term papers, 
essays, book reports 

-any constructed materials 
such as paintings, musical 
scores, diagrams, maps, de- 
vices, models 



Process measure 

Procedures for comparing : 

-any spoken responses such as 
those made in dramatizations, 
public speaking, foreign 
language, technical discussion 

-some psychomotor performance 
such as the form used while 
stroking keys of a musical 
instrument, throwing a base- 
ball) pouring liquid chemicals 
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Of the learner's performances below, indicate which involve 

0 

"product” .criterion measures and which involve "process criterion 

measures. Place "Prod" or "Proc" in the boxes to the right. 

0 

The learner will: 

.prepare a circuit diagram, ^ . 

.write a short poem about childhood, | 

..question a fellow student without 
arousing hostility, * | 

.open a student government meeting with 
the pledge of allegiance. | 

The first two yield tangible products which will require 

procedures for rating the product. The last two require establish 

ing procedures for observing and rating learner performance as it 



occurs . 

In the next list of criterion measures below, indicate in the 



same fashion whether each is a product or process measure. 
Rating scales or scoring sheets for: 

1. Interpretive reading of an eulogy 



2. a contour map drawn from given 
elevations . 

3. a half-gainer dive 

4. a flow diagram 

5. signalling a right-hand turn 



c 



You were correct if you indicated the following answers: 



1. Proc 

2 . Prod 

3 . Proc 

4 . Prod 

5. Proc 
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BEACTIVE AND NON-REACTIVE CRITERION MEASURES 

Let's look, now, at the second scheme mentioned earlier which 
can help generate a broader variety of criterion measures. 

The terms " reactive and non-reactive" suggest that some 
measures may be reactive, i . e . , may change t he natural 
response we would be interested in prompting and measur- 
ing. 

For example, students often speak of the desirability of "psy- 
ching out" the instructor so as to know what position to take during 
class discussions or in writing answers to test questions. There- 
fore, common standard testing procedures such as quizzes and exami- 
na.*tions represent examples of reactive measures, at least to the 
extent that the students know the kinds of responses previous in- 
structors have looked for. 

To illustrate, suppose that an instructor in a Health Science 
class asks his students to list some reasons for (or against) the 
pj^^ictice of smoking marijuana. If the students are aware that the 
instructor's scoring procedures will reward only those reasons given 
for not smoking marijuana, then this measurement via a test question 
represents a reactive criterion measure since it leads the students 
to modify the responses they might otherwise have given. 

Examples of less reactive measures in this context might be: 

.submitting a "contemporary word meanings" questionnaire 
v:!'ere students are asked to match words common to the 
marijuana smoking sub-culture with their meanings in 
everyday English. 

.Observing and recording how students choose sides in a 
proposed debate on the topic "Should Marijuana Be Legalize 
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Suppose a speech instructor is concerned that students 
use particular speech practices which he is trying to strengthen. 
Which of the criterion measures below is likely to be least reac- 
tive, and result in the more accurate data? 

.Listening to a 60 second impromptu talk in class, 

.Listening to a classroom discussion of a contro- 
versial topic, among peers, in a "buzz session." | | 

It is more probable that the discussion with peers will be 
least reactive and that speech habihs will emerge more naturally 
in the "buzz session" than in the speech to the class. 

In the situation below, select the criterion measures which, 
you feel would be least reactive: Please place a check mark by those 

you choose. 

1. Instructor decides to determine the extent to which his 
class is doing outside reading of reference materials. 

.he tallies the number of hands raised in response to 
the question "How many were able to do some reference 
reading?" 



.he tallies the number of enrollees names on a library 
check-out sheet for reference materials. 



2. A typing instructor hopes that students will use the 
correct form for letters of correspondence. 

•he assigns students to submit a letter (to a hypothetical 
company) requesting an appointment for an employment 
[_ interview; letters are obtained and scored. 

•he asks that students write and mail a "thank you" 
letter to a guest speaker who visited the class; letters 
are obtained and scored. 



You were correct if you selected the second option in both cases 
above. Both of the first alternatives are more likely to make the 
students awar.' of the desired behavior sought by the instructor. 
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SETTING STANDARDS OF MINIMUM ACCEPTABLE PERFORMANCE ' 

So far in this paper you have been urged to consider a broad- 
er variety of criterion measures which would enable you to assign 
a performance index (e.g./ a score) or performance indices (e.g., 
multiple scores) to observations of student behavior. Thus when 
we have obtained a series of scores or performance indices follow- 
ing instruction, we are then, (and only then) in a positron to de- 
cide whether the instructional program has measured up to expecta- 
tion or whether the program needs revision. Making the decision 
as to what you will set as a standard of acceptable performance on 
the part of the learner is called setting minimum perf ormance stan- 
dards . 

For example, suppose that an instructor has decided that he 
wants learners: 

to identify common insects by -their- scientific names. 

He obtains 10 pictiires of common insects and tells his students 

"Look at each picture carefully and then write in the 
blank on your answer sheet the scientific name of each 
insect. You will be given one point for each name that 
is correct according to the identifications given in your 
textbook . ” 

This instructor then decides that the instructional program 
shall be d'^emed successful if: 

80% of the students correctly write the scientific names 
of 90% of the insects pictured. 
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Now, please reread the above and decide which statement is the 
instructor's overall objective , which is the statement of the cri- 
terion measure , and which statement describes minimum performance 

standard . 

You probably recognized that the last phrase represented the 
minimum performance standard and the first phrase represented his 
general objective . Obviously, the middle phrase describes the 

criterion measure . 

MULTIPLE CRITERION MEASURES 

In attempting to obtain evidence of the learner's ability to 
perform the behavior described in, the instructional objective it is 
desirable to employ several different kinds of criterion measures. 
The use of multiple criterion measures should produce a more reli- 
able estimate of performance than any single measure. Especially 
in those cases where a measure is a) difficult to obtain, b) pre- 
dictably reactive, or c) obtainable only in some distant future, 
multiple criterion measures are called for. It is desirable that an 
instructional plan contain several measures of criterion performance 
even though only one or two are ultimately used. 

This paper has presented two schemes for categorizing multi- 
ple criterion measures. It should be noted that other schemes have 
been proposed. One particularly helpful approach has been attached 
to this paper as an Appendix. The examples provided will assist you 
by suggesting usable criterion measures as you develop your own ob 
jectives and measures. 




12 . 



i 



SUMMARY 

r 

This paper has provided instructions so that you might learn 

to accomplish the following tasks: 

•to identify those measurement procedures which are suit- 
able as criterion measures (i.e., those which measure 
the behavior called for in the objective) . 

•to distinguish between product and process criterion 
measures . 

•to distinguish between reactive and non-reactive criterion 
measures . 

.to distinguish between an overall objective, a criterion 
measure for that objective# cind a minimum perfoirmance 
standard to indicate satisfactory attainment of the ob- 
jective. 

.to derive a number of criterion measures meeting the a- ' 
bove standards, given an objective. 

Please complete this practice exercise so that an assessment 
can be made as to the extent to which this booklet has been success- 
ful. The minimum performance standard for this paper predicts that 
85% of you will be able to attain a perfect score. If the standard 
set for your performance is not met, the booklet will be modified. 

CRITERION EXERCISE 

I, Please select the alternative most appropriate as a criterion 
measure for the objective. 

Objective: Learner will decide if a given author is "for” 

(or against) a given political viewpoint. 



\ 






f 

t- 




% 


\ 


Ih 





Alternative A, 
Alternative B, 



Alternative C< 



Learner will read the author's political essay. 

Learner will describe in writing the author's 
position and note whether (or not) the author 
is favorable to a political action. 

Learner will tell why the author takes a cer- 
tain political stand. 
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II. Adjacent to each criterion measure below, indicate which are 
product (prod) and process (proc) measures. 

A rating scale or scoring sheet for: 



A. 


safe handling of a power saw ^ _ 


1 A. 


B. 


1 

a typed inter-office memo ^ 


1 B. 


C. 


choral reading in unison 


— 1 c. 


D. 


a soldered wire 


n D. 


E. 


f ■ * 

a set fiompleted equations 


1 E- 


F. 


an historical time-line ^ 


^ F. 


G. 


posture while writing ^ 


1 G. 



III. For each of the following situations, select the criterion 
measure which is least reactive: (Note: Lea^ reactive.) 



Situation : 



Instructor wishes to determine how many of his 
students enjoy solving problems in mathematics 



Possible 
(Please check 
least reac.) A. 




B. 



Criterion Measures > 

He counts the number of students who raise their 
hands in response to the question, "How many of you 
enjoyed solving these problems today?" 

He collects and tallies responses to a questionnaire 
he submits which asks if students enjoy solvxng math 
problems. 



C He provides several free-choice activities and notes 
how many students elect to solve math problems from 
the range of alternative activities presented. 



Situation: 



instructor wishes to find out It students thinic 
it important to keep their workshop machinery in 
good condition. — 
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Possible Criterion Measures: (Please Check Least Reactive) 

A. He counts those who mention this in their lists of 
"Things to do to improve our classroom workshop. ” 

B. He counts those who nod their heads in agreement when 
he suggests that they keep their workshop machinery 
in good condition. 

C. He observes and counts those learners who volunteer 
to clean up the workshop and repair the machinery. 



>' IV. Label each of the phrases below as being either an overall 
objective (0 ) , the criterion measure (CM) , or the minimum 
level of acceptable performance (MLP) . 

Write 0/CM/MLP 
in boxes 

A. "Three-fourths of the learners will correctly sequence 
I all of the steps needed." 

B. "Learners will be able to correctly sequence the 
steps necessary to diagnose malfunction in a T. V. 
set." 



C. "Given a scrambled list, learner. will unscramble the 
list and be scored "credit" if all steps are in 
correct sequence." 

END OF CRITERION EXERCISE 

answers to the criterion exercise should have been: 

B 

A. Process 

B . Product 

C. Process 

D. Product 

E . Product 

F. Product 

G. Process 

C. 

C. 

A. MLP 

B. 0 

C. CM 



Your 

I. 

II. 



III. 

IV. 
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APPENDIX 

CRITERION MEASURES TO DETERMINE LEARNER ACHIEVEMENT^ 



Standardized criterion measures 

1. Standardized achievement and ability te sts , the scores from 
which inferences are made regarding the extent to which cog- 
nitive objectives have been attained. 

2. Standardized rating scales and check lists for judging the 
quality of products in visual arts , crafts / shop activities, 
creative writing, exhibits for competitive events, cooking, 
typing, letter writing, fashion design, and other activities. 

3. Standardized tests of psychomotor skills and physical fitness. 

4. Standardized self inventories designed to yield measures of ad- . 
justment, appreciations, attitudes , interests, and temperament 
from which inferences can be formulated concerning the possession 
of psychological traits (such as defensiveness, rigidity, ag- 
gressiveness, cooperativeness, hostility, and anxiety) . 

Teacher-made criterion measures 

5. Achievement tests (objective and essay) ; the scores on which 
allow inferences regarding the extent to which specific in- 
structional objectives have been attained. 

6. Rating scales and check lists; for observation of classroom 
behaviors; performance levels of speech, music, and art; man- 
ifestation of creative endeavors, personal and social adjust- 
ment, physical well-being. 

7. Questionnaires; frequencies of responses to items in an ob- 
3 ective format and numbers of responses to categorized dimen- 
sions developed from the current analysis of responses to open- 
ended questions. 

8. Interviews; frequencies and measurable levels of responses to 
formal and informal questions raised in a face-,to-face inter- 
action. 



^Adapted from A Paradigm Inv olving Multiple Criterion Measures for 
the Evaluation of the Effectiveness of School Programs . Written 
by Newton S. Metfessel (University of Southern California) and 
William B. Michael (University of California, Santa Barbara) . 
Offset, 10 pp. 
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9. Self-evaluation measures : student's own reports on his per- 

ceived or desired level of achievement, on his perception of 
his personal and social adjustment, and on his future academic 
and vocational plans. 

Peer nominations ; frequencies of selection or of assignment 
to leadership roles for which the sociogram technique may be 
particularly suitable. 

11. Projective devices ; such as casting characters in a class play, 
role playing, and picture interpretation based on an informal 
scoring model that usually involves the determination of fre- 
quencies of specific behaviors, or ratings of their intensity 
or quality. 

Miscellaneous criterion measures 

12. Absences ; full-day, half-day, part-day and other selective 
indices pertaining to frequency and duration of lack of atten- . 
dance. 

13. Anecdotal records ; critical incidents noted including fre- 
quencies of behaviors judged to be highly undesirable or highly 
deserving of commendation. 

14. Attendance ; frequency and duration when attendance is required 
or considered optional (as in: club meeting, special events, or 
off-campus activities) . • - 

15. Books ; number checked out of library, numbers renewed, numbers 
reported read when reading is required or when voluntary. 

16. Changes in program or in teacher ^ requested by student ; fre- 
quency of occurrence . 

17. Choices expressed or carried out ; vocational, avocational , and 
educational (especially in relation to their judged appropri- 
ateness to known physical, intellectual, emotional, social, 
aesthetic interests, and other factors). 

18. Dropouts ! numbers of students leaving school before completion 
of program of studies. 

19. Grouping; frequency and/or duration of moves from one instruc- 
tional group to another within a given class grade. 

20. Library card possessed or not possessed ; renewed or not renewed. 

21. Peer group participation ; frequency and duration of activity in 
what are judged to be socially acceptable and socially undesirable 
behaviors . 
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22. Skills: demonstration of new or increased competencies such 

•as those found in physical education/ crafts / homemaking/ and 

the arts that are not measured in a highly valid fashion by 
available tests and scales. 

23. Tape recordings and video-tapes : critical incidents are 

corded and other analyzable events amenable to classification 
and enumeration. 
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ARRANGING INSTRUCTIONAL ACTIVITIES 



Rita Johnson 

You have already selected your instructional objectives and 
developed multiple criterion measures for a target group of 
learners. In order to arrange an instructional sequence it is 

necessary now to specify in detail what is needed to accomplish 
these objectives. 

Such a set of specifications is typically produced by advanced 
planning engineers and then passed on to others who are directly 
responsible for actual technical-production efforts. Instructors 
however, usually prepare their own comprehensive lesson plans or 
sequential units prior to instruction. Once such a unit has been 
carefully outlined, any other instructor should be able to teach 
from such specifications with minimum difficulty. 

Therefore, the purpose of this booklet is to help you in the 
preparation of an organized instructional sequence which will later 
be produced and tested. 

Careful attention to this booklet should help you to do the 
following: 

. 'recognize a variety of instructional variables which, when 
applied, could alter or determine your instructional se- 



quence ; 
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. • organize an instructional unit which incorporates all as- 

pects of the instructional design thus far, including 
specifications of what is needed by the instructor and 
the learners before instruction can occur; 

. arrange instructional activities and material into a se- 
• quential pattern which utilizes a broad variety of instruc- 

s ’ 

tional variables. 



COMMON INSTRUCTIONAL VARIABLES 

Below are some of the more commonly accepted variables which 
research findings suggest affect learning. It may be that not all 
or any of these are necessary for effective teaching. It remains 
for the instructor to select from as wide a range of alternatives 
as possible in hopes that these will be effective. Once he in- 
cludes them in his instructional sequence, he is in a better posi- 
tion later to empirically test whether or not these variables in- 
deed make a difference. 

1. ADVANCE ORGANIZER . An advance organizer is analogous to a pre- 
face in a text. It is a statement by the instructor which pro- 
vides the learner with a kind of cognitive structure or "set'* 

* 

such as a key phrase which signals the important topic yet to 
come. 

One example would be a paraphrased objective, such as, 
"Today we are going to learn about the ways in which. .. "or"Our 
immediate problem is...." and "Let’s find out if..." 
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2. PERCEIVED PURPOSE. This refers to questions or statements by 
the instructor which prompts the learner to perceive the im- 
portance or purpose of the instruction to follow. Emphasis 
here is on the need for the students to perceive why or how 
what they are about to do is purposeful or meaningful to them. 

For example f the instructor may ask^ "How will solving 
this problei.^ help us when we try to...?" or "Hov/ can this be 
useful to you on the job?" He might also note, "This will make 

you more efficient when you try to..." 

3, MULTIPLE ELICITORS . These refer to a series of questions and/ 

or statements designed to produce an intended series of learn- 
er responses which are at successively higher levels of task 
(e.g., re: Bloom’s Taxonomy ) . Such statements prompt the 

learner to perform in a series of ways which gradually lead 

to the response specified in the objective. 

For example: 

"Do you recall that objectives are statements of intended be- 
havior changes in the learner?" (Recall) 

"Stated differently, does this mean that teaching consists 
of changing the learner’s behavior?" (Comprehension) 

"What, then, do teachers change when they hope to affect a 
student’s attitude in the classroom?" (Application) 

"In the film you are about to see, how many attitudinal behav- 
ior changes can you identify?" (Analysis) 

"Now write an instructional plan which includes at least three 
attitudinal changes in your objectives." (Synthesis) 

Not all elicitors produce a desired or intended response. If 

the objective calls for "discriminating fact from fiction," an 
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elicitor which asks students to "listen* to so...- .naterial may or ^ 
may not be very helpful, depending upon whether or not “listening" 
behavior is part of a series which will lead to "discriminating" 

behavior* 

Advance organized, statements of purpose and elicitors are 
motivational in intent. In the space provide place an 0, P or 
E beside each quote to show if it is an Organi zer, statement of 

Purpose f or EXicihor * 

Objective: Learner will calculate prices of items which 

are to be discounted. 

1. "HOW will learning to do this help you as a consumer?" 

2. "What is the price of this washing machine which is to 

be discounted by one- third?” 

3. "Now let us look at the problem of determin- 
ing discount prices at a market*" 

4. "Calculate the prices of the following four items." 
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Answers ; 1 . 



P because it forces the learner to analyze 
why the learning will be important to him. 

2 , E because it elicits the intended behavior- 
al response. 

3, O because it organizes or provides a "set” 
in advance of instruction. 

4, . E because it elicits an intended response. 

4. PROMPTS OR CUES . These refer to hose hints or aids the in- 
structor provides in order to help the learner accomplish a 
desired response. These may be provided when the learner is 
recognized as having difficulty or when the instructor wants to 
insure success, such, "Watch out now. I may try to fool you. 
and "Pay close attention. This problem is complex." 

Some common examples of cuing include; 

" Remember yesterday we said that..." 

"If this i s true, what about...?" 

"When you hear the poem, listen for..." 

"As you watch, notice whether or not..." 

"Can you find three instances where...?" 

5, PRACTICE. This refers to activities which give the learner 

opportunity to practice the specific behavior .to be accomplished 

« 

by the end of instruction. Without opportunity to practice 
relevant or "appropriate" behavior, the learner is not as likely 

to perform well on the criterion test. 

As in the case of multiple elicitors, there are multiple 
opportunities to practice, each of which successively approx- 
imate the desired response specified in the objective. 












Practice, then, should be consistent with the objective and 

• 

the criterion measures. Generally speaking, the relationship 
is as follows: 

Ob-iective ! A statement that students are expected to 

perform in a certain way. 

pyar»t,ire : Students are given several chances to perform 

in that way. 

ryii-pyinn Measure* Students are tested on whether or not 
' they can perform in that way. 



For the following objective which two learning activities are 
the best examples of "relevant” practice? 

Objective: Learner will label common insects with their 

scientific names: 



1. Reading about the names given to common insects 

2. Watching a film on insects 

3. Writing the names of insects under pictures of insects. 

4. Matching pictures of insects with printed names on 
cards • 



Answers: 3 and 4. . 

6. SMALL STEPS . Programmers are foremost advancers of the need 
for small steps to accomplish learning. How small each step 
should be depends in part upon the complexity of the task, 
difficulty of the content, ability of the learner, and so forth. 
If the steps are too small, boredom may occur. However, lengthy 
chunks of content can be tedious and difficult. 
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7. PACING. Rslated to th© problem of making the learning task 
simple yet challenging, is the problem of whether the sequence 
is too fast or too slow. Pacing refers to the rate at which 
the learner must go tlirough the instructional material or 

activities. 

8. GRADUATED SEQUENCE . Learner behavior can be arranged in suc- 
cessive steps until the criterion performance is reached. Like- 
wise, the activities and content of instruction could be grad 
uated in sequence until they approximate the behavior and con- 
tent implied. in the objective. 

A number of logical sequences have been suggested. None, 
however, have proved significantly better than any other. In 
fact, some studies have demonstrated that when frames in a 
programmed sequence are scrambled, learning does not signifi- 
cantly decrease! 

Commonly used sequences have been: 

from simple to complex 

from facts to generalizations 

from concrete to abstract 

from practical to theoretical 

« « 

from meaningful to unknown 
from past to present 
from present to future 

9. KNOWLEDGE OF RESULTS . This refers to responses given back to 
the learner which tells him how well he has just performed. 

When a student gets feedback on when and why he is right or wrong. 
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he can take steps to improve and is more likely to repeat 
the correct response next time. 

Some instructors tell learners why they were incorrect, 
then give them an opportunity to try again. In this way 
learners are getting additional practice until they make the 
appropriate response and can be told they are correct. 

10. POSITIVE REINFORCEMENT. This refers to anything the instruc- 
tor does which tends to increase the probability that the 
. student will perform in the same way and repeat his response 
in the future. Certain instructor behaviors serve as rewards 
or "reinforcers** for the learner. Such reinforcers are likely 
to elicit similar responses next time. 

For example, an instructor's smile, nod of the head, or 
softening of the voice could serve to reinforce some students. 
Knowledge of having correctly performed is also a reinforcer. 

However, what is reinforcing to one learner may not be to 
another. It is therefore important to examine the type of re- 
inforcer used and determine its appropriateness for a learner's 
particular age, sex or background. 

Thus far you have examined a broad variety of instructional 
variables which could be incorporated into your instructional 
sequence. Before these can be arranged, however, it will be 
helpful to detail all other aspects of your instructional system 
thus far. 
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PREPARING THE UNIT OF INSTRUCTION 
There are many types of format which could be used to detail 
specifications for a unit. A simple one can be found in APPENDIX 
I. You should refer to this form now before you read an explana- 
tion of each of the categories listed below. 

After filling in the details at the top of the form (if they 
apply to you) plan to complete the following: 

1. Prerequisites . 

List those skills, concepts and attitudes which you assume the 
learner possesses upon entering the instructional environment. 

2. Objectives . 

List here the objectives you intend to accomplish as a result 
of your instructional environment. 

3. Criterion Measures . 

Pnr each obiective , list the criterion measures which will enable 
you to assign scores to observations of student behavior. 

4. Learner Activities . 

For each objective, list the activities you intend to provide 
the learner so that he may learn to perform intended responses. 
At tliis stage, they need not be in any graduated sequence. 

Since this is only a worksheet or mock-up of specifications, 
brainstorming appropriate activities may prove to be very use 

ful. 

At all times the criterion to use for selecting such ac- 
tivities is whether or not they will help the learner to acquire 
practice needed for accomplishing the objective. 
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5.- Related Content . 

For any activity there may be potential reading material which 
will help the learner to respond in appropriate ways. List 
here the exact chapters, pages and names of relevant books, 
nagazines or articles which have reference material for use 
by you or your students during the activity. 

6. Media and Material 

Later you will be, helped to select and produce a variety of 
instructional media and materials. At this point, you may want 
to list audio-visual aids or media which you suspect are mean- 
ingfully related to the activity or the content of the objective 
If you know of specific films, filmstrips, slides, photos, re- 
• cordings, kits, transparencies or aids, this is a good time to 

list them. 

Again, the criterion to use for selecting media is whether 
it will help the learner practice what is needed to accomplish 
the objective. In this way, the learner stands assured of per- 
forming well on the criterion test and you stand assured of be- 

ing successful. 

Brainstorming may help to produce a rich range of 
alternatives that may prove useful. These can always be 
discarded later. Remember that if an activity or medium is 
selected which bears no relation to your objective, several 

alternatives are still possible: 

a. a new objective could be added, 

b. an existing objective could be modified. 



c. the activity or medium could be altered to fit the 
• objective, or 

d. the activity or medium can be dropped. 

Self-checking procedures are needed to re-establish your 

original intentions. Frequently, especially after many changes 
in your chart have taken place, it is wise to stop and see 
whether or not all aspects of the chart are internally 
consistent with your original objectives . A checksheet is 
provided for this purpose in APPENDIX 2. . 

Since it is assumed that the completion of the list will 
take time, it is suggested yoU leave the chart and checklist 
for now and continue finding out about ways to sequence your 
unit once the chart is completed." 

SEQUENCING THE INSTRUCTIONAL EN7I0RNMENT 

There is more than one way to arrange a unit of work into 

♦ 

a sequential pattern. The two suggested here involve: 

a. arranging the content or lecture material, and 

b. arranging the learners’ activities or behavioral 
responses. 

In the first case, the lecture or expository material 
is simply written out in some logical order (e.g., from 
simple to more complex) and divided into smaller chunks or 
lengths at appropriate intervals. Each "chunk" then becomes 
a sort of "frame" to which may be added a variety of 
instructional variables. 

For example, to each "frame" might be added: 

a. an advance organizer 

b. a statement to provide purpose 



c. elicitors 

d. some prompts to cue the correct response 

e. an opportunity to respond (i.e., practice) and 

f . immediate feedback to the learner as to how well he 
responded. 

It is not necessary that these variables be repeated with 
regularity in each frame. For example, intermittent prompts 
(i.e., cuing heavily at first, then gradually reducing cues until 
they are no longer needed) is very effective. Withdrawal of 
prompts allows the learner to approximate more and more that 
behavior which is most like the criterion test performance 
(which will offer him no prompts at all) • 

At any rate, a certain degree of purpose, opportunities for 
appropriate practice and feedback of results would probably 
increase the effectiveness of your sequence a great deal« 

A second way to arrange a sequence is to list the activities 
for each objective on your list in graduated sequence (e.g., from 
simple to complex re: Bloom’s Taxonomy ) . 

Again, each activity then becomes a "frame” or chunk of time 
around which relevant content and a variety of instructional 
variables may be incorporated. 

For example, one simple rule- could become: ^ 

a. Tell the student what he*s to do (prompt’ or elicit). 

b. Let him do it (provide opportunity for practice) 

c. Tell him if he did it (knowledge of results) 

It now remains for you to select and develop appropriate 
media which may considerably alter this tentative sequence. 
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The test below asks you to recognize the meaning of some 
Important terms. If the author is successful , 90% of the 
readers will attain a score of 80% or better. 



In the space provided, write the letter of 

the staitement that best describes each term: 

1. ' Advance Organizer a. A statement designed to 

produce an intended 
response in the learner. 






2. Perceived Purpose 



b. What the instructor does 
so that the learner is 
likely to repeat the 
response next time. 



• 3. Elicitors 

4. Prompts or Cues 

5. Practice 

:6. Small Steps 

7. Pacing 

8. Graduated Sequence 

9. Knowledge of Results 

10. Positive Reinforcement 
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c. 



d. 



e. 



f. 

g- 

h. 
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A' statement which aids or 
helps the learner to be 
successful in his response. 



A measure by which a score 
can be assigned to judge 
a learner response. 

Arranging activities and 
related content into 
manageable "chunks” of 
learning. 

A statement which provides 
a "set” for learning. 

Speed with which learners 
must grasp the material. 

A statement telling a 
leArner whether or not he 
is responding correctly. 

When the learner knows why 
or what he's about to do 
is meaningful to him. 

Arranging a pattern of acti 
vities so that they succes- 
sively approximate the 
behavior in the objective. 
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k. An activity which provides 
opportunity to engage in 
the intended behavior. 

l. A statement of what learner 
is to do as a result of 
instruction. 
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Below are the answers to the criterion test. If you did not 
attain a perfect score you may want to reread the section, 

COMMON INSTRUCTIONAL VARIABLES again. 



1. f 

2. i 

3. a 

4. c 

5. k 

6. e 



8. j 

9. h 

10. b 



It is suggested that you now do the following: 

1. Fill out the form in APPENDIX 1. Let others help you 
brainstorm ideas . 

2 , When completed, use the checksheet in APPENDIX 2 to check 
for internal consistency with original objective. 



3. For one objective, write an outline 6f a script for a 
sequence of instruction which utilizes at least five 
instructional variables. Begin by arranging either the 



content material or learner activities . 
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APPENDIX 2 



CHECKLIST 



CHECKLIST: 

1. Do the students still possess the prerequisites 
needed as entering behaviors? 

2. Are your criterion measures still measuring 
.. .behavior implied in the objective? 

. 3, Are all learner activities still relevant or 
helpful in accomplishing the objective? 

4. Is all content still related or of help in 
accomplishing the objective? 

5. Are the media and materials still related clearly 
to some objective? Will their use help students 
perform well on the criterion test? 
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SELECTING AND DESIGNING METHODS AND MATERIALS 



Rita Johnson 

The purpose of this booklet is to provide you with step-by-step 
procedures for the design of a self-instructional system. It will 
also provide you with guidelines for the identification and selec- 
tion of instructional methods and media. 

Method/Material Selection 

The selection of methods and materials 
should be based upon the definition of 
objectives, criteria, and content or 
task analyses. 

Unless you are required by unchangeable circumstances, you should 
rigorously avoid, any commitment to the types of method or media you 
will use until you have completed analysis of your objectives and 
criterion measures. 

This is because no one best method exists for any objective. For 
example, APPENDIX 1 lists a broad range of instructional methods, sev- 
eral. of which might be appropriate for any one objective. 

To illustrate, in the space below, write two objectives from your 

unit : 

1 . - 

2 . 

Now turn to APPENDIX 1 and select three alternative methods which 
could help accomplish each objective. Consider only those you do not 
typically use and write them below; 

Obj. 1; a. b. 



c. 



If you found it easy to consider some methods of instruction which 
were new to you, you have already demonstrated a willingness to keep 
an open mind in searching for alternative methods and materials. 

MATERIALS SEARCH 

It is a fairly safe assumption that you will save money and time 
if you can find commercial or otherwise prepared materials which fit 
your objectives. Even if you can*t find materials that apply directly, 
you can often borrow bits and pieces, and/or make alterations. 

The first step of a materals search is to develop a set of de- 
scriptors for the subject of instruction . The ERIC (Educational Re- 
search Information Center) thesaurus is recommended as a useful tool 
to start your building of the descriptor list. These descriptors 
facilitate yoiir combing through the countless media catalogs much like 
those descriptors you use in your library research. 

You have probably had experience obtaining material on preview, 
^st commercian companies are quite willing to mail review copies or 
materials on approval. Rental libraries are a good source of materials 
idiich are otherwise unavailable. 

When you preview materials you should work from a standardized 
form to assure uniform judgment . If the material appears promising, 
take notes, or in the case of a film or video tape— make an audio 
tape of the soundtrack. This will enable you to integrate the content 
while you are waiting for the material to be acquired. 

There are three possible outcomes that may be expected of your 
material search, (l) You may find the right material at the right 
level in the right quantity. (2) You may not find enough or any 



appropriate materials, thus you must begin originating your own. 

(3) You may locate an overabundance of good material, in which 
case, you must produce further criteria for limiting material selec- 
tion. 

The two things to remember in material 
selection are: (l) Let your objectives 

guide your material search. (Don*t start 
with a film, textbook or some other form of 
media you like and then justify its use.) 

(2) Make your search as broad as possible. 

Hopefully you have already selected two or three methods and have 
located several material packages to accomplish your objectives. The 
major reason for the selection of these methods and materials should 
be that they are most effective for accomplishing the objective . 

The final selection must consider some practical considerations . 
These would include not only costs, but also the time allowed to gain 
the objective, the number of students who will take the vinit, the 
available classrooms and the instructional machinery. 

Can you recall the major reason for selection of 
methods and materials, as well as three practical con- 
siderations? Write them below: 

Major Reason : 

1 . 

Practical Considerations t 

1 . - 

2. 



3. 
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Answers : 

Ma.1or Reason ; 

1. They help you accomplish the objective 
Practical Considerations ; 

1. cost 

2. time available 

3 . student numbers 

4 . available classrooms 

5 . machinery available 



PRINCIPLES OF SELECTION 



The first principle in media selection is to con- 
sider the instructor as a medium of instruction * 

In fact, instructors are the most expensive media in an instruc- 
tional system. At least 60^ of the operating budget of most school 
districts goes to certified salaries* 

Consider the efficient and professional use of the instructor as 
a medium of instruction. First, he is the most flexible of all media. 

He can change pace, rate, even content in response to the learner’s 
condition. Second, he is the most responsive of all media. He can 
feel and react uniquely to unique questions. He can build pyramids of 
associations. He can tie content specifically to an individual learner’s 
interests. He can adjust content to an individual student’s ability 
level. Note that these unique advantages of the instructor are lost 
when the class size is too leirge for two-way communication between 
learner and instructor. 

The second principle of materials selection is to 
consider all of the parts which are required in order 
to have a self-contained unit. 
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By "self-contained" we mean the complete set of materials and 
media which by themselves are necessary and siifficient to achieve an 
instructional objective. As an illustration, an overhead projector 
and a set of transparencies are not sufficient. They support a 
teacher in the presentation of information. The teacher’s cost must 
be included in describing such a unit. On the other hand, a tape and 
tape recorder, linked with a filmstrip and projector may be sxifficient 
to achieve an objective and therefore would be a "self-contained unit." 

The third principle in determining media is to trans- 
late media costs into a cost per student hour for compari- 
son purposes . 

This is a necessary step in convincing budget-minded administrators 
and trustees of the utility of an instructional approach. APPENDIX 2 
offers a model for making cost comparisons, assuming that you are faced 
with a choice between an overhead-instructor set versus a tape-filmstrip 
set. 

The cost comparison in APPENDIX 2 illustrates principle one above. 
The unit cost of an instructor plus supporting media is almost always 
higher than for self-contained media because the instructor is the most 
expensive element in the system. However, teachers are a superior medium 
of instruction, if they utilize their skills property. 

The fourth principle in media selection is to pur- 
chase instructional equipment only if you have sufficient 
material to use the equipment twenty or more hours per 
week . 

This principle relates to having enough operationsil hours at your 
institution to reduce the cost per hour to a low figure. The more you 
use a piece of equipment the more efficiently it is being used. This 
leads to the fifth and final principle. 
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The fifth principle in media selection is to 
purchase media in terms of the ob.jectives of major 
curriculum units * 

To achieve a sufficient base for the justification of equipment 
the aims and objectives of more than one class may have to be con- 
sidered. The media applications with high initial costs such as 
Audio-Tutorial, Dial Access, Computer Assisted Instruction and most 
forms of closed -circuit television can only be justified if they are 
of use to the entire curriculum. But, if considered on this basis 

these general media approaches may be the cheapest form of media. 

ORIGINATING MATERIAL 

One factor which will influence your development of original ma- 
terials is availability of production facilities on the campus. In 
general the smallest educational institution can afford the equipment 
to produce the following materials ; 

Visuals 

slides Overhead transparencies 

35mm filmstrips 8mm motion pictures 

Sound 

Audio tapes 

None of the above materials requires expensive facilities. Produc- 
tion can be done by well-trained students or the instructional staff. 

The production of video tapes requires more expensive equipment and a ’ 
higher J.evel of training for the staff. But if this medium is used 
properly the results are well worth the effort. 

This booklet will direct your design efforts towards a media system 
which has proven to be effective when used in an educational setting, the 
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Audio -Tutorial Approach . This direction should not limit your method 
selection since it contains all of the media forms which could be used 
individually in the classroom. That is, it already includes the use 
of an instructor in small and large group discussions, yet also pro- 
vides for self -instructional audio -tutorial lessons. 



The Audio -Tutorial Lesson 

An audio -tutorial lesson is an audio tape com- 
bined with either a workbook, a filmstrip, a slide, 
or an 8 mm film loop, or some combination thereof. 

The student takes the lesson to a study carrel where 
he controls the equipment. Combined with the lesson, 
there may be an exercise or an experiment. The audio 
tape controls the entire lesson, and the student is 
in complete control over all media sources. 



CHARACTEEISTICS OF AN AUDIO-TUTORIAL LESSON 



1. It can integrate several forms of material such as films, slides, 
filmstrips, etc. 

2. It is self-contained . 

3 . It may be stored in a form which can be used repeatedly for many 
years. 

4. It is individualized in that the student can take the lesson at 
his own convenience and proceed through a series of lessons at 
his own pace. 

5. It is only the machine element of a ” man -machine ” system . The 
necessary miin portion of the system consists of instructors in- 
teracting with students in small discussion groups. The instruc- 
tor makes the information meaningful to the student by associating 
it with his interests and needs, and by dealing with his indivi- 
dual learning problems. 

I’he Audio-Tutorial lesson is under the control of the student. He 

may proceed at his own pace, repeating parts of the lesson as many times 

as necessary. He may take the lesson any time the laboratory is open. 

The audio-tutorial lesson is especially useful when comparing, analyzing. 



o 
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problem-solving, or drilling is required. On the other hand, the in- 
structor is now free to meet with individuals, smaU groups and large 
assemblies in order to tutor, clarify, amplify, evaluate, and en- 
courage or inspire. 

Let us eummarize your progress ot this point. You are now develop- 
ing and creating a self -instructional, unit which will need to be tested 
shortly with a few students . You may wish to direct yo\ir efforts to- 
ward developing a tape -controlled system, such as the audio-tutorial. 
Instead you may wish to develop a series of self -instructional activi- 
ties, none of which would be controlled by audio-tape. Either system 
should not re struct your selection of methods or materials in any 
fashion. In fact it should increase the possible alternatives. 

PREPARING AN AUDIO-TUTORIAL OR JJELF-INSTRUCTIONAL LESSON 
Step I: From your unit of instruction, select a large segment of 



content or lecture material which is clearly related to 
one ob,*jective and write a sequential script . Be certain 
to utilize a few instructional variables which might im- 
prove learning (e.g., advance organizers, perceived pur- 
pose statements, elicitors, prompts, appropriate practice, 
small learning steps, knowledge of results). 



Step II: Write down on one half of the sheet of paper what you say 
and on the other half what you would write on the black- 
board. 



Narration 



Blackboard 




on the blackboard 



”You show them what you 
want to reinforce . ” 




step III: Analyze what you have said in the lesson for potential 
visualization* 

(a) First look at your nouns and adjectives. Are you 
describing something which you could visualize by 
picture, a graph, or an outline? 

(b) Examine your verbs and adverbs. Are you describ- 
ing an action, an activity or a motion which could 
be shown rather than explained? This examination 
for implied motion should be done carefully. For 
instance, when you describe causality you are de- 
scribing motion* When you describe associations 
on multiple levels, you are describing motion. 

When you discuss dimensions, you can often explain 
a fourth dimension by the use of motion. The in- 
troduction of motion into a lesson generally has 
positive influence on the student's receptivity. 



Narration 

"You could tape your lecture 
and photograph what is shown 
on the blackboard.” 



Visual 

Slide : Instructor writ- 

ing at blackboard while 
speaking into a micro- 
phone connected to a tape 
recorder. 



(c) Circle all nouns and verbs in your script which 
can be visualized. 

(d) Indicate the form and content of the visualization 
as in the example. 

(e) Change your narration to reflect and support your 
visualization. 

Step IV: Examine your script for possibilities of student activity 
or experimentation. Remember in the Audio -Tutorial lab 
the student will not be tied to the chair. He or she can 
get up to examine displays, operate a piece of machinery, 
or take some other action which will result in discovery 
and understanding* These activities should be implied in 
your post test. The student will generally learn more if 
he has an opportunity to actively practice what he is to 
learn. 
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Narration 


Action 


"At the sound of the chime 


Students will stop the 


you will stop the tape and 


tape. Students will 


circle the nouns and verbs 


circle nouns and verbs 


in your script which can be 


to be visualized. 


visualized. When you are 


Students will turn the 


through turn the tape on. " 
-Chime- 


tape back on. 



step V: Kow that you have visualized and activated your script, ex- 

amine once again the places you might utilize instructional 
variables to increase learning effectiveness (i.e.. Are your 
learning steps small enough? Do you need to add prompts? 
Are students finding out if they responded correctly? ) 

Step VI: Now examine the remaining element, the sound * The first 
consideration is the narration; in person or on tape, you 
can do it yourself or engage someone with a trained voice. 

Another consideration is supplementary sound. Are you re- 
ferring to an object which has a natural sound that could 
be heard on the tape? Can you use more than one voice, a 
guest speaker or a dramatic dialogue? Would excerpts of 
music or choral reading further your objectives? 

Examine your script and indicate in the margin any sound 
possibilities . 

Step VII: After you have reworked your script to reflect appropriate 
activities, the visuals, and the supplementary sound ele- 
ments, it is tine to start assembling materials. Note that 
your visuals fall into three broad classes; 

(a) Graphic Material ; (Similar to what you write on the 
blackboard.) The preparation of these materials 
should be done carefully since you are creating a 
permanent record. Use color whenever possible to 
reinforce learning points. The use of professional 
lettering equipment or a graphic artist is desirable 
but not necessary. A careful freehand will probably 
be clearer to the students than your normal writing 
on the blackboard. These graphic visuals can be 
photographed into filmstrips. 
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(b) Photographs of Actual Objects 

Here again it is best to start with slides which 
will later be copied onto a filmstrip. Hith 
modern camera equipment it is quite easy to make 
either slides or filmstrips. You may have had 
experience in the use of a slide camera for family 
photographs. If you don*t have technical assist- 
ance> don’t be afraid to do your own photography. 

(c) Motion Pictures 

It is also possible to consider two classes of 
motion picture visuals. The graphic visuals are 
animated sequences such as found in cartoons. 

The technique can be very effective in explaining 
subject matter, but is more difficult to use than 
live action. Live action is easily photographed 
in. 8mm. Again you may be familiar with the equip- 
ment mid techniques through your family photograph- 
ing experience. 

Step VIII: Assemble your visual materials into a set of slides and/or 
a 8mm motion pictui-e film. Now select a student from your 
target population. Try to choose a slow learner since he 
will teach you the most about your writing ability. Talk 
the student through the lesson, allowing him to ask ques- 
tions freely. Make a tape of your narration, and the 
questions the student asks so you can study it for later 
revision. If you have made extensive changes, select 
another student and repeat the process. Repeat the 
process as often as necessary to the point where you do 
not have to make explanations supplementary to the script . 
Your multi -media package is now already for empirical 
testing. However, if you„are preparing a lesson which is 
to be audio-tutorial you must complete the final step. 

Step DC: Assembly the final Audio-Tutorial lesson. Prepare a film- 

strip from the augmented slide set and tape narration. 

There are two ways to tape the final narration. The profes- 
sional way is for you to record from a sound booth while 
observing a student who is listening and working through the 
lesson. This will give you cues for proper pacing. 

The other way is slightly less effective, but much simpler. 
Borrow a good tape recorder, suid make the recording in a 
quiet room. The problem with the latter technique is that 
you must make a conscious effort to avoid sounding pedantic. 
It is difficult to keep a natural tone in your voice when 
speaking to a microphone. Recall that you are talking to 
an individual student and not a class! 



o 
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The purpose of this booklet was to help you select and origi- 
nate a broad-ranged methods/materials system. If the author is suc- 
cessful, 90 ^ of the readers will attain a score of 80 ^ or more on 
these true -false items ; 

1. The Audio-Tutorial system is the only system that indi- 
vidualizes instruction and integrates a variety of media. 

2. For each type of objective there is probably one best 
method. 

3 . Finding commercial materials which fit existing objec- 
tives will save you money and time. 

If. One way to insure uniform Judgment of materials to be 

selected is r,o work from a standardized form. 

5 . Cost^ time, size of class, and available machinery are 

considered to be practical considerations in selecting 
media. 

6 . An instructor should search for a film he knows and likes 
even if it does not fit his objective. 

7 . The cost of the teacher must be included in describing a 

multi-media unit. 

8 . The most flexible and responsive mediiua of instruction is 

the instructor. 

9 * student controls the pace of an audio -tutorial lesson. 

10. The complete Audio-Tutorial System includes use of instruc 

tors interacting with students in small and large discus- 
sion groups. 



NOTE; 



PLEASE RETURN REVISION DATA SHEET — LAST PAGE 
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The answers to this test follow and can be verified by reread- 
ing the booklet: 



1. 


P 




F 


2. 


P 


yi- 


T 


3- 


T 


/ 8. 

9. 


T 


4. 


T 


T 


5. 


T 


10. 


T 



As a further activity, carefully do the following: 

1. Select one or two objectives from your unit. 

2. Produce a sequential script (including visualization, 
activation and sound) which is designed to accomplish 
the objectives selected above. 

3» Provide for small steps, prompting, practice and knowl- 
edge of results within the script. 

4. Prepare to empirically test your system on from 1-4 
learners who will be able to help you refine the sys- 
tem before its final implementation with the total 
learner group. 




APPENDIX 1 



METHODS OF INSTRUCTION 

The principal methods by which instruction is provided: 

Comparative Analysis - A thought process , structured by the teacher, 
employing the description, classification, and analysis of more than 
one system, group, or the like so as to ascertain and evaluate simi- 
larities and differences. 

Demonstration - An activity in which the teacher or another person 
uses e^camples, experiments, and/or other actual performance in order 
to illustrate a principle or show others how to do something. 

Diagnosis - The continuous determination of the nature of learning 
difficulties and deficiencies, used in teaching as a basis for the 
selection - day-by-day or moment -by -moment - of appropriate content 
and methods of instruction. 

Directed Observation - Guided observation provided for the purpose of 
improving the study, understanding, and evaluation of that which is 
observed. 

Discussion - An activity in which pupils, under teacher and/or pupil 
direction, exchange points of view concerning a topic, question, or 
problem in order to arrive at a decision or conclusion. 

Drill - An orderly, repetitive learning activity intended to help de- 
velop or fix a specific skill or aspect of knowledge. 

Experimentation - An activity involving a planned procedure accompanied 
by control of conditions and/or controlled variation of conditions to- 
gether with observation of results for the purpose of discovering rela- 
tionships and evaluating the reasonableness of a specific hypothesis. 
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Field Experience - Educational work experience, sometimes fully paid, 
acquired by pupils in a practical service situation. 

Field Trip - An educational trip to one or more places where pupils 
may study the content of instruction directly in its functional 
setting, e.g., a trip to a factory, newspaper office, or fire depart- 
ment. 

Group Work - A process in which members of the class, working coopera- 
tively rather than individually, formulate and work toward common ob- 
jectives under the guidance of one or more leaders. 

Laboratory Experience - Learning activities carried on by pupils in 
laboratory designed for individual or group study of a particular sub- 
ject-matter area, involving the practical application of theory through 
observation, experimentation, and research, or, in the case of foreign 
language instruction, involving learning through demonstration, drill, 
and practice. This applies also to the study or suet and music, though 
such activity, in this instance, may be referred to as a studio ex- 
perience . 

/ Lecture - An activity in which the teacher gives an oral piesentation 
of facts or principles, the class frequently being responsible for note 
taking. This activity usually involves little or no pupil participa- 
tion by questioning or discussion. 

Manipulative and Tactile Activity - Activity by which pupils utilize 
the movement of various muscles and the sense of touch in order to 
develop manipulative and/or perceptual skills . 

Modeling and Imitation - An activity, frequently used for instruction 
in speech, in which the pupils listen to and observe a model as a basis 
upon which to practice and improve their performance. 
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PyS'C'b ice Session •• All activi'ty in which pupils have opportunity 
to put into practice those skills and understandings previously learned 
through other instructional activities. 

I^pblem -Solving - A thought process structures by the teacher and em- 
ployed by the pupils for clearly defining a problem, forming hypo- 
, thetical solutions^ and possibly testing the hypothesis. 

I^p^ramed Instruction - Instruction utilizing a workbook, or mechanical 
and/or electronic device which has been “programmed” to help pupils 
attain a specified level of performance by (a) providing instruction in 
small steps, (b) asking one or more questions about each step in the 
instruction and providing instant knowledge of whether each answer is 
right or wrong, and (c) enabling pup»ils to progress at their own pace, 
^o^ect - A significant, practical unit of activity having educational 
value, aimed at one or more definite goals of understanding, and involv- 
ing the investigation and solution of problems. 

Reading - Gathering information from books, periodicals, encyclopedias, 
and other printed sources of information, including oral reading and 
silent reading by individuals. 

Recitation - Activities devoted to reporting to a class or other group 

t 

about information acquired through individual study or group work. 
Role-Ilay - An activity in which students and/or teacher take on the 
behavior of a hypothetical or real personality in order to solve a 
problem ani j^ain insight into a situation. 

Seminar - An activity in which a group of pupils, engaged in research 
or advanced study, meets under the general direction of one or more staff 
members for a discussion of problems of mutual interest. 

o 
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Sensitivity Training - An activity in which a group and a trainer meet 
to self-consciously examine their immediate feelings and perceptions 
about themselves and each other, in order to gain skill in authentic 
communication, leadership, behavioral flexibility or social sensiti- 
vity. 

Shopvork - An activity emphasizing skill development through experience 
in woodwork, metal work, or other industrial processes and procedures. 
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PART 1 



MODEL FOR MEDIA SET COST COMPARISON 



1 . Equipment Costs 


Instructor 

Overhead 


Tape 

Filmstrip 


1 . Initial Cost of 


equip. 


$150.00 


$300.00 


2 . Amortized for 
6,000 hours of 
operation. 

(30 hrs./wk X 4o wks/ 
yr X 5 yrs = 6,000 
hrs.) per hour 


.025 


.050 


3 . Maint enanc e /hour 


.010 


.020 


2 . Material Cost 






a. Visuals - 30 frames @ 


.30 frame @ 


9.00 




30 frame 
filmstrip 




1.50 


b. Narration cost tape 




1.50 


c. 20^ of les»?on is 


revised each x 5 yr«. 


9.00 


3.00 


d. Total cost material 


for 5 years 


18.00 


6.00 


e. -—6,000 hrs. 


» 

0 

0 


.001 


Instructor 


Tape 


Media Costs 


Overhead 


Filmstr: 


3. Total material and 


.038 




equip, cost per hour 


.071 


4. Instructor time per hr. 


6.00 




5. Total cost per hour 


6.038 


.071 


6. Per student hour ® 30 


students/hour 


.201 




@ 1 student/hour 




.071 



EXPLANATION ON NEXT PAGE 
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PART 2 
EXPLANATION 



The first step is to compute the cost per hour of the equip- 
ment. This requires you to estimate the useful life of the equip- 
ment in terms of hours of operation. We estimated that all of 
the equipment we are comparing would last five years or for 6,000 
hours of operation. We also estimated that repairs and maintenance 
for each piece of equipment would cost $.01 per unit per hour. 

The second step is to compute the cost of the material. We 
are not counting the preparation time since we assume that would 
be the same for both types of material. We assume that there are 
thirty visual frames in the lesson, that overhead transparencies 
cost $.30 per frame and that a thirty- frame filmstrip costs $ 1 . 50 . 
The instructor will supply the narration that accompanies the 
transparencies but a tape costing $ 1.50 is required for the tape- 
filmstrip set. 

For purposes of comparison we assume that the basic material 
sets will be good for five years. To be safe we assume that 20^^ 
of the set will be revised each year. At the end of five years 
the material will have undergone a 100^ revision cycle. This total 
cost can be divided by the 6,000 hour base period, for a per hour 
cost. 

The per hour costs of the equipment and the material is added. 

The instructor *s time must be added to the overhead costs to 
make it a self-contained set. This cost cannot be amortized since 
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the instructor must be present every time the lesson is given. 

The tape and filmstrip are a self-contained set; the instructor s 

presence is not required. 

5, The total set cost per hour is the total of the equipment, ma- 
terial and instructional costs per hour. 

6. The final step is to divide the total numiber of students who 
will benefit from the lesson at one time by the total cost per hour 

which gives the cost per student hour. We assume that the instruc- 

^ / 

tor-overhead lesson is given to thirty students at a time. The 

filmstrip tape lesson is given to only one student at a time. 

Note that the initial cost of equipment in a tape -filmstrip 
set is twice that of the overhead. However, the actual unit cost 
is approximately one-third since the instructor must be added to 
I the cost of the overhead. 
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REFUIING THE INSTRUCTIONAL SYSTEM 



Rita Johnson 

This booklet presents a procedure for the empirical testing 
and the revision of an instructional unit^ component^ product or 
program. It deals with the need to successively refine the initial 
draft of a proposed unit of instruction. 

Lesson plans^ units ^ courses of study, or any reproducible 
sequence of instruction designed to produce observable changes in 
learner behavior must undergo empirical test and modification. 
Successive try outs and revision prior to actual instruction signi- 
ficantly heighten the probability of success when instruction is 
implemented. 

The instructor who seeks to attain his objectives will want 
to test the material out on small sets of learners (2 to 5 students) 
prior to its full-scale implementation. He will examine learner 
responses carefully and make appropriate modifications in the ma- 
terial to increase the likelihood of learner achievement. He will 
then try the modified material with a new set of learners and make 
further refinements until a specified level of performance is 
achieved by all learners. 



2 






ERIC 



Careful attention to this booklet will result in your being 
able to: 

• list three categories of problems likely to be found in 
instructional material duriiig an empirical try out 

• cite examples of specific instructional improvements 
which can be made in light of learner responses received 
during an empirical try out 

• describe five procedures for obtaining learner responses 
during an empirical try out so as to eliminate superfluous 
or confusing characteristics of the instructional material. 

******************* 



COMOH PROBLEM AREAS 



In the early stages of developmental testing, an instructor can 
actually study the individual learner as he learns. Obtaining feed- 
back from the student, of course, is not new. Even when lecturing, 
an instructor may ask for questions, leading to further examples, 
restatements, and so forth. This process may lead to a complete re- 
vision of the original lecture. 

What is relatively new, however, is the notion of closely ob- 
serving an indivj.dual learner working in the early stages of a pro- 
gram design. Rather than taking lengthy chunks of finished material 
into the classroom, the instructor should test small parts of his 
unfinished design with a single learner. IdeeJ2y the feedback from 
learner to instructor begins early, proceeds work by word, element 
by element, and ceases only when the instructional researcher has 
achieved his goal— an effective teaching tool. 

What is vital here is the notion that the learner is going to 
teach you. The learner cannot fail, for if he does not get where you 
want him to go, your material has failed. You must therefore try 
something else. In fact, in the absence of anything better it might 
be best to trust your intuition and vary your approach until you are 
successful. Then edit the material and try it on another learner. 
Make whatever changes are necessary for your material to take care 
of both students. After three or four tries, each with a different 
learner or set of learners, you will have instruction that is more 
likely to accomplish its objective than before. 

Many writers on this subject insist that the first draft should 
fail to achieve the predicted goal if it is to make full use of data 



from the learner being tested. If. the instruction works completely 
on the first trial, the learner has been deprived of an important 
opportunity to contribute to the design of your instruction. Typi** 
cally, the first draft vTill elicit lear<ier responses of a 70^ accu- 
rate nature with approximately 70^ of the try cvA learners. 

During your trial runs you may find problems being uncovered 
which fall into three fairly distinct categories : 

• : a need for rewording or rearrangement of 

material to improve the sentence structure, vocabulary, 
explanation, instruction or question. 

• Attending ; a need to heighten interest to gain attention, 
arouse curiosity, provide a challenge, and thereby lessen 
distraction, boredom, or ape.thy. 

• Instructional -variables ; a need to a.lter the presentation 
to improve the pace, sequence, organization, size of learn- 
ing step, use of prompts, opportunities for pra.ctice, or 
reinforcement . 

****** -K- ************ 

Three categories or problems have been cited which are likely 
to be uncovered while empirically testing your instruction. lihat 
are these three categories? If you wish, go back and reread the 



three definitions. 



Below, write what you consider to be an example of a specific 
revision you might make to solve a problem in each ca^tegory: 

-• Clarity: 



• Attending: 



♦ Instructional -Variables : 



Examples of ways to improve clarity might be to reword the 
instruction, add a clarifying example, simplify an explanation, 
change the vocabulary, shorten the sentence or add a definition. 

Problems in attending might be lessened by adding some humor, 
eliminating a distraction, providing a sense of purpose of impor- 
tance, adding an arousing or attention-getting segment. 

Instruct ional -variables may be improved by providing more 
practice activities, adding cues or prompts, rewarding the learner 
more frequently, shortening or lengthening the number of steps in 
an activity, slowing up the pace, or reorganizing the sequence of 



activities. 



Below are over a dozen modifications you might make when you 
try out and revise your own instructional program. Indicate with 
letters which ones you believe to be directed toward solving prob- 
lems of clarity ("C"), attending ("A"), or instructional-variables 
("I"). 

1. make instructions clearer 

2 . attract the student *s attention 

3* reword the question 

4. change the vocabulary 

5. shorten the number of steps in aui a 
activity 

I 

6. simplify the explanation 

7 . give a clarifying example 

8. speed up the pace 

9 . add some humor 

10. add some extra prompts 

11. add a practice exercise 

12. provide a sense of purpose and impor- 
tance to the activity ^ 

13 . reward the learner after the activity 

14. rearrange the sequence of activities 



Answers : 



!• C 6 . C 11 . I 

2. A 7. C 12 . A 

3. C 8 . I 13 . I 

C 9 . A l4. I 

5* I 10. I 

****************** 
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MULTIPLE PROCEDURES FOR OBTAINING LEARNER RESPONSES 



Instructional technologists have developed a variety of proce- 
dures for obtaining learner responses to each element in instruc- 
tional material to eliminate superfluous or confusing portions . A 
combination of these approaches plus others should be employed. 

Regardless of which combination of procedures is used, the in- 
structor should adopt a tutor -like approach with his learners and 
employ a tape recorder to record the empirical test. In this way, 
later distortions due to memory loss or selective perception are 
likely to be lessened. 

During the try out, at least five different methods for obtain- 
ing learner responses might be tried: 

(1) Error-rate approach ; the basic concept here is to collect 
responses to each item, question, or element in the process, and then 
analyze and modify those items which produce error or do not result 
in the desired learner response. 

By locating such errors, the instructor is provided with clues 
as to where appropriate modifications of the material are needed. For 
example, if the student responds with, ”I cannot answer that question, ” 
the instructor may decide that there is need for some additional 
prompts and practice exercises, or simply that the question is poorly 
worded. 

(2) Interview technique ; this technique consists of asking for 
reactions or comments during or following a lesson. The instructor may 
not necessarily want to lead the learner and invite opinions as to how 
a specific element might be revised. Yet, when the learner does make 
an error, the instructor uses that error as information together with 



candid comments the learner might make to modify the materials being 
developed. 

For example, the learner might add, *‘I guess I wasn’t paying 
much attention, " in which case the instructor may decide the problem 
was one of lack of interest. This might lead to the inclusion of ma- 
terial designed to arouse curiosity or challenge the learner. 

( 3 ) Diagnostic - criterion test approach ; this techniq.ue uti- 
lizes a diagnostic test, in addition to the regular criterion test, 

in an attempt to determine the degree of attainment of all prerequisit 
tasks or behaviors assumed to be needed prior to accomplishing the 
criterion test. Diagnostic errors suggest which specific sections of 
a program need to be deleted, expanded or reworked. 

For example, in developing a mathematics program, the instructor 
may wish to find out which elements of the program need expansion and 

r 

more thorough treatment. Analysis of the errors on a diagnostic test 
will tell him which tasks or concepts need to be taught more care- 
fully before he can expect overall mastery of the objectives. 

(4) Latency of response approach ; this technique is ba.sed on 
studies which indicate that delay in responding to items is an effec- 
tive means for identifying item deficiency. Latency of response (even 
when the correct response is eventually given) tells the educator 
when the learner is experiencing difficulty and when parts of the' ma- 
terial need to be revised to make more of a contribution to the out- 
come performance. 

If, for example, a learner indicates he vzants to "think about it 
for awhile" the instructor has a clue as to where a modification of 
the material is needed. The student may be confused, the material 



presented may be ambiguous or the material may be too difficult. 



( 5 ) Black-out rate procedure : this is a method proposed for 

eliminating extraneous parts within a lesson. It simply requires 
deleting or covering over (with paper, black crayon, or ink) all 
learner activities or expository material which you believe might 
not be essential in producing a desired response. In this manner 
the instructor has a useful guide to determine how much might be de- 
leted without increasing errors. 

Therefore, if you have a hunch that certain material or activiti 
are superfluous, you might try deleting it to see whether it makes a 
d.ifference. It has been demonstrated that over 50 percent of exist- 
ing material may be blacked out and errors are not significantly in- 
creased. This means you might delete half of your instruction and 
cut your teaching time in half, without decreasing any learning! 

Thus far, five approaches for obtaining learner responses have 
been discussed. These are; l) error-rate, 2) interview, 3 ) dia 
nostic test, 4) latency of response, and 5) black-out. 'VJhich pro 
cedure do you think is being used in each instance below? Write the 
procedure next to each of the examples. 

a. The learner spontaneously says, ”I don’t like this.” 

b. Instructor asks for additional comments about any 

aspect of the lesson. 

c. An instructor of English composition gives a 

vocabulary test. 

d. Several learners respond that they’re "think- 
ing it over." 

e. The instructor elects to omit the first half 
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f. Instructor notes that three learners didn’t follow 
a specific instruction. 

g. Instructor decided not to tell his favorite Joke. 

h. Learners complain that an assignment was d\ill. 

i. Instructor notes that when she asks a question no 
learner responds immediately. 

J. Of 10 learner responses given in a lesson, four 
are Judged as inappropriate. 

******************* 



J 




The answers are 



6t • 


Interview 


f. 


Error -Rate 


b. 


Interview 


g* 


Black-Out 


c. 


Diagnostic Test 


h. 


Interview 


d. 


Latency of Response 


i. 


Latency of Response 


e. 


Black-Out 


0* 


Error -Rate 




*********** 


* * 


****** 
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CONCLUSION 

The purpose of this booklet was to provide you with some pro- 
cedures for improving the initial design of instructional material 
and activities. Empirical testing of your proposed program in the 
form of several try out and revision cycles was suggested to i-e3.p 

you refine and upgrade the effectiveness of your material. 

t 

It is apparent, of course, that such developmental testing can 

« 

be costly and time-consuming. In fact it would be difficult to empiri- 
cally try out every lesson, activity or component prior to its im- 
plementation. The saving in learner time on an empirically tested 
unit of instruction, however, may more than repay the investment of 
time and energy spent on testing and revising that unit. 

The tester does more than observe, make a tape-recording, and 
take notes. He must be sensitive to overt expressions of thinking 
on the part of the students. He must notice gaps, predict con- 
fusions, sense response failures, rewrite new items on the spot and 
elicit honest responses. He must let the learner teach him without 
being defensive. This humility and responsiveness to criticism comes 

* 

hard to many instructors. Such a person must be able to learn from 
the mistakes of one learner, capitalize upon them when faced with a 
new learner, and consistently improve upon the material’s effective- 
ness. 

Since only a few learners are being tested at this early stage, 

they are not a very representative sample of those in the actual in- 

♦ 

structional setting. Can one really learn Mich by observing only one 
learner? Amazingly, one is much better than none.* You probably will 

er|c 



14 



find that observation of even one learner can significantly improve 
your first draft of a program. Hopefully, you will have an opportu- 
nity to make three or four empirical try outs and revisions and each 
cycle of modification should move you toward your goal of some level 
of mastery attained by all learners being tested. 

******************* 
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If the author of this booklet has been successfia, of the 
readers should now be able to take the following criterion test and 
attain a perfect score. 

1. Below are five procedures for eliminating superfluous or confusing 
elements in instructional material. Write the letter of the state- 
ment that best describes each procedure: 

Deleting superfluous material 
to see if needed. 

Determining length of time 
taken before learner responds. 

Obtaining spontaneous , candid 
comments from learner. 

Determining where mistakes are 
being made most frequently. 

Determining if student has 
mastery over certain skills. 

2. Three general categories of problems* are likel 5 ^ to be uncovered dur- 
ing an empirical tryout of instructional material. Give two ex- 
amples of each. 

a. Clarity: - — 

b. Attending: .. 



Error -Rate : 



Diagnostic-Criterion Test: 3. 



Latency of Response: C. 



Interview Approach: 



D. 



Black-Out Approach: 



E. 



c. 



Instructional -Variables : 



l6 



3 . List six examples cf material revisions which might be made in 
your instructional material after an empirical test. 

3 # • , - - - ^ - - — 

b. 

c. — 

d. 

e. - 

f. - 



r 



i 



Answers : 



1 . 



2 . 



3. 



out 



1 . 



2 . 



1. Err or -Rate: D 

2. Diagnostic -Criterion Test: E 

3 * Latency of Response: B 

4. Interview Approach: C 

5 . Black-Out Approach: A 

See Pages k and 5* 

See Pages 4, 5 3-nd 6. 

For an important additional activity, you might consider carrying out 
the following steps : 

Try out your program on from one to four learners • In light of 
learner response, make necessary modifications in your draft. 

Try out the revised program on another small set of learners and 



make necessary revisions. 

3 . Try out your refined program on a third set of learners and refine 
once again. 

Finally, it would be worth your time to go back and examine the 
combination of methods you used and the problems that emerged during 




each empirical test. Did you try to improve clarity, attending, or 
instruct ional -variable s ? VJhat revisions did you make? Are you mov- 
ing toward your original system goal of a specific level of learner 
mastery over the instructional material? 

******************* 
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